2  ^ 

^V°-‘  oS-  *"”"CV"y.>-‘'-<*' 

a  t  " 


‘‘‘j^  A  O/'o  •  )i-‘'‘ Jif^  * 

'^.V_  /^s5L" 


'  O 
'*> 


. 

^  ^  <3  csi  <«  _ 


r  '<K,  >* 

^  £)  N  O  ^  jk.  ^ 

*  ‘  ' 


o  ,  .C> 


«r  n  ^ 


'  'V  -^  *• 

*  ^ov  * 

;  ^°^<l^,  * 

^ <11^  *^OnO'^ 

<J5r  ^  5  *  0>  <x  1 

-  y^  ^ 


«7  J\\  83  /h,  ^  ’y^ 

•  -  ° 

Lia  ong^  ^  ^  ^  ^  ^  ^  ^ 

'^.,  \„/’  ‘"jfc*- V„'i‘'  - 

V  ,  v  ^’■ « ft  %  <5tt  j^.‘>  lLl^^  >~.  ^ 

Xw/  -.Ilf  /  •<!-''X.  X^.- 

o#i.-»  J>  „„„  "'o  » >■'*  ^ 

.^o  »  '  utn^  I  ao  '#w  v  j  r^’X'^x  tk^  **  \_^  -  •■ 

« 


v<b 

to  "j^  n 

■'X'wr-'v-  < 


V  r-)  ik*.  j*  >>vSit^  a’ 

*  %  G,' 

O  g"  ®«»  *  %-C?^^ 


V-P 


>ov^" 


t*  Vw  ’ 


^  ffjT  TV  ^  ^  ^  ^  9;;. 

I  ^  o^fet 


c^^-> 

o 

-^6  ^ 
^  y^  r 


,0'  cOX^*^ 

»  Cj  *  ■•' 

0“  “ 


8^  n  ^ 

^  -» 
o 


^  ^  .  V.=^  w  .  _ 


\A  . .  ^  <A.  *>^0 « ^cy  ^ 


o 

"•J'i.  y\ 

'*^>  ^»ONO'5\<hV'  9^ '•»,.. 

^  ^’^j.  ^■%'  ^'■^^^/v,%9(>  A- 

-»^iyAx  G-i- 


s<xs»**>-g9>  “'‘°  ■^.1-'"*  °* 'Hx 


<5^S^  ^ 

O 


»  ^ov 

y  ^  <f‘ 

V  "=>«<>’ . 


*  ’•■**  ^toN 


« ^"/V°  ’ 


f  ^  r^'^'  ’>"  X" 

>'^4^'  ‘*'»i>N0’*v«^  9) .  ^ 

c  <(  iV  >.  ,  ^v  «*  ■<  ■^  O  >, 

<r  ^  Wf^iStet^'^  ^^'.’■!K\//l!’'r,  9f 

<t? 


o 


Vx 

^  A.^  'C>  ^ 

^  -V 


,  x-»'  .0""  '*& 

-  A  .ik 


i^oso’^ 


V  % 

qv  co«'=«.%r'''‘""i^ 

%  'Arf?  « 

o  ^  ^Q-y  «  ^  V'O^  o 

A  *1  'S^Vii^  t«>9KtVr»  Am-.  <1 


^OV  <» 

. o,  -H,  ‘>°  . .  *  ^  ^\,  ♦  o^Pv"*’ 

^  'V>'6‘  o  r  c55'^'n 


cX-'^'-Xo 


^Q. 


C'"’  •ifcX'"'“:>X’, 


%X  ,cp^ 

o 

t 

ibJ?, 


^OV  « 


it  TV 


X.^rxA  yv* 

*  ^v  ^ 

%>^J.<9  X  9 


\  rfi 

'  y*  -  v»s»<y-  -, 

,  »ono'5 

V^  'S 


* 


>  *  ■«  ^-  A* 

*‘-'*/<f/“‘‘’-*o'$‘'^c,o«'=C 

%  'V  cP  *  - 

;  -f^d*  "' 


’  * 
O 
to 


.4’^r. 


c,- 


*  LI,  '^*■'0  ,  X'*'*  -S^  ^0.\ 

<l'^  »*  .■#J«>a'''.  'O 

V  <-,^^;  o 

-1 X  <» 


^  ov  <» 


^  1).  s 


r>%.  <iy  -h- 

i  \<^  “SSmt 


‘"-^'^"■\lw/  /’\  iW/  I 

6^'^^°’'“  v'V" "  ’  ‘'"'►^V°  ’ 

S^  *  *  -^o*  °  g^ia, »  '^ov  « •*  vd*  ® 

‘  -'ft"  tf  j'M-  ^^V-^'’-'' 

i>.  a  *■  <  y 


O 


'Ta  ^12®  ^  o  w/^w  to  ^ 


(!> 


^O, 


\o^' 


►  .Tjv  '-(;)  .iT^  -Jv -r  '1*^  /-V  li 


'*'  '^o>  » 

.“V  -— 

■^ov^ 


<P«C?3  C3  Kj 


%  o 

O 


rCr  f.oNG 


'o  '.w.'  o'  X  '.w."  '•■ 

;  °  "J%:  %«v^  =mk\  ' 

r  £5>J?n.  1  mi^  o  .».VA  -  ,<?,J'^  n 


>5^'"  V\W/  /\  %W/ 

>.<■  *•">/<>/“  •  ’•'^  o^*^®'’"“«#'\  ^ 


’ONO'3^^  ^O.^it 


'^V  ^  D.  ^■^cy  ° 

^  AN  <* 


^1  ^ 
'o 

^  O 

V  "<5*^ 

o  <P  c> 


tt'^  K^ 

,  y  n 
<«  ^ 

’X'W/yV^- 

“  j»  1*^  -A^  ^  Avr/Vi^  ^ 

A^V!«Zrv^r. 

A 


^  <,  A  ^ono'^  C^ 

4*^ -j^.  .<v  ^’■^^%(?/v,  ^ ' 


:  ipx  ’- 

^  -O 


X'"’  ’i-*-”  LiL'-V-“’  v<.<  ■  »?■ 


%'\  ^^'•' 

*  '^‘o’  °  , 

(O'  .‘C'*  V  ■  ■‘^'’■^•1^*?'V”'  *'  d«'‘’ V‘  •  •  'J 

^  '^ov  <*<0^:  ^OV^ 

>P<i*  ^  ^  ^  ^ 

.  ..  s  -  r  V  V  °^"' 

*"o  V 


ff  n  tv 


-  t 


^  'j'er  ° 

*  ° 

o 


^  »OKO'’  ,  *o»Pl'  *»n*ViP 

*  w  :^m\  \c^  “"^fe: 


\6 

t  % 


fo  ^uyk  ;« 

’>\^  *  ofV'-  *  oS^ 

%•  AV/h,’®  '^V;<88tete\  A  A'-'’^vA  ^ 


>J 

'.  'ov*  . 

w  ’ 

,  4/  o 

*  ^  s> 


%  S 

*  A\  \ 

'  X  .0° 


■^fs- 


>.o« 


.Cs  .0 


Medical  Team  Training— Technical  Review 


Agency  for  Healthcare  Research  and  Quality 
Advancing  Excellence  in  Health  Care  •  www.ahrq.gov 


PATIENT 

SAFETY 


This  report  may  be  used,  in  whole  or  in  part,  as  the  basis  for  development  of  clinical  practice 
guidelines  and  other  quality  enhancement  tools,  or  a  basis  for  reimbursement  and  coverage 
policies.  AHRQ  or  U.S.  Department  of  Health  and  Human  Services  endorsement  of  such 
derivative  products  may  not  be  stated  or  implied. 

AHRQ  is  the  lead  Federal  agency  charged  with  supporting  research  designed  to  improve  the 
quality  of  health  care,  reduce  its  cost,  address  patient  safety  and  medical  errors,  and  broaden 
access  to  essential  services.  AHRQ  sponsors  and  conducts  research  that  provides  evidence- 
based  information  on  health  care  outcomes;  quality;  and  cost,  use,  and  access.  The 
information  helps  health  care  decisionmakers — patients  and  clinicians,  health  system  leaders, 
and  policymakers — make  more  informed  decisions  and  improve  the  quality  of  health  care 
services. 


DoD  Medical  Team  Training  Programs:  An 
Independent  Case  Study  Analysis 


Prepared  for: 

Agency  for  Healthcare  Research  &  Quality  (AHRQ) 

540  Gaither  Road 
Rockville,  MD  20850 
www.ahrq.gov 

Office  of  the  Assistant  Secretary  of  Defense/Health  Affairs 
(TRICARE  Management  Activity) 

U.S.  Department  of  Defense 
Skyline  5,  Suite  810 
5111  Leesburg  Pike 
Falls  Church,  VA  22041 
www.ha.osd.mil 


Contract  No.  282-98-0029,  Task  Order  No.  54 


Prepared  by:  ut 

American  Institutes  for  Research 
David  P.  Baker,  Ph.D. 

J.  Matthew  Beaubien,  Ph.D. 

Amy  K.  Holtzman,  M.A. 


OF 


AHRQ  Publication  No.  06-0001 
May  2006 


This  document  is  in  the  public  domain  and  may  be  used  and  reprinted  without  permission  except 
those  copyrighted  materials  noted,  for  which  further  reproduction  is  prohibited  without  the 
specific  permission  of  the  copyright  holders. 

Suggested  citation: 

Baker  DP,  Beaubien  JM,  Holtzman  AK.  DoD  Medical  Team  Training  Programs:  An 
Independent  Case  Study  Analysis.  (Prepared  by  the  American  Institutes  for  Research  under 
Contract  No.  282-98-0029,  Task  Order  No.  54)  AHRQ  Publication  No.  06-0001 .  Rockville,  MD: 
Agency  for  Healthcare  Research  and  Quality.  May  2006. 


♦ 


< 


11 


Acknowledgements 


This  document  was  developed  from  work  supported  by  the  Agency  for  Healthcare  Research 
and  Quality  (AHRQ)  and  the  TRICARE  Management  Activity  of  the  U.S.  Department  of 
Defense  (DoD).  The  American  Institutes  for  Research  (AIR)  would  like  to  thank  Dr.  James 
Battles,  Ms.  Heidi  King,  ETC  Beth  Koshin,  USAF,  and  CPT  Glenn  Merchant,  USN,  for  their 
comments  and  advice  during  the  preparation  of  this  report.  AIR  would  also  like  to  express  its 
thanks  to  the  developers  of  MedTeams™,  Dynamics  Outcome  Management®,  and  Medical  Team 
Management  for  allowing  us  to  attend  their  training  and  providing  unlimited  access  to  important 
training  materials.  Access  to  this  information,  in  part,  made  this  case  study  possible.  We  want  to 
commend  the  efforts  of  Dynamics  Research  Corporation,  Crew  Training  International,  the  U.S. 
Air  Force,  AHRQ,  and  the  DoD  for  seeking  to  improve  patient  safety  by  promoting  better 
teamwork  in  health  care.  We  believe  that  efforts,  like  the  ones  described  in  this  report,  will 
ultimately  lead  to  improved  patient  safety  in  the  DoD  and  elsewhere. 


Ill 


\  ’ 


...... 


I . 


i  -f 


^’1?  V 


\i  '  '■  .  .  ■■  ...r*  ..  -la  pi 


i*^.  l^•^-'..v.  *A<^.  >* 

I/I  Ai»v*'  ■'•'■-  '  ■  ■  ... 

*...:,'  r.n '-’ 


*<• 


•%  r 


h 


■»  .  [:  i\'i*  ’  •  »'•  ’XCJJ 


.J^ 

r  '  ‘ V, 


-.! 


I  *  *  *^,  s 


*  *■ 


tr  * 


*M 


■  fA* 

■  I.'*  i 


.  ••’v.V' M' ^  ; 

.  i:'f''n;,v,,.....;.v-p',. 

•  -<"  k 

*.  '  ■  i‘m  v**^ 

A  ^ 


■r. 


\A 

V  •;  a  .'<• 


^ ...  :' 

I*  •  * 


.'1  ■'  '.I  *"*c' 


i 


I 


Abstract 

Crew  resource  management  (CRM),  a  form  of  team  training  that  was  developed  for  military 
(and,  subsequently,  civil)  aviation  has  been  adapted  for  use  in  health  care  settings.  Several 
different  CRM-derived  medical  team  training  programs  have  been  in  use  within  the  Department 
of  Defense’s  health  care  system  and  in  some  civilian  hospitals.  The  Agency  for  Healthcare 
Research  and  Quality  awarded  a  contract  to  the  American  Institutes  for  Research  (AIR)  to 
evaluate  three  DoD-sponsored  medical  team  training  programs — MedTeams™,  Medical  Team 
Management  (MTM),  and  Dynamic  Outcomes  Management®  (now  called  LifeWings'^^).  AIR 
had  previously  conducted  a  review  for  AHRQ  of  the  research  literature  concerning  medical  team 
training. 

To  conduct  this  evaluation,  the  researchers  first  reviewed  student  and  instructor  guides, 
slides,  and  other  audiovisual  materials  that  the  course  developers  provided  to  them.  Second,  the 
researchers  attended  and  observed  the  classroom  portion  of  each  of  the  three  programs,  collected 
pretraining  data  on  student  experiences  and  expectations  and  new,  independent  posttraining  data 
on  student  reactions  to  the  programs.  Finally,  trained  AIR  staff  conducted  one-on-one  interviews 
with  MedTeams  and  MTM  instructors. 

The  results  suggest  that  all  three  programs  possess  several  desirable  characteristics,  such  as 
using  active  learning  techniques  and  offering  training  to  interdisciplinary  health  care  teams. 
Nevertheless,  each  program  also  had  a  number  of  limitations.  For  example,  not  one  of  the 
programs  was  based  on  a  comprehensive  pretraining  needs  analysis  and  participants  had  limited 
opportunities  to  receive  structured  practice  and  feedback  on  critical  teamwork  skills.  Based  on 
their  observations  from  the  three  case  studies,  the  researchers  set  out  the  framework  for  a 
successful  medical  team  training  program,  and  recommended  further  actions  to  improve  and 
support  future  medical  team  training  programs. 
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Executive  Summary 


Introduction 

For  fiscal  year  2003,  the  Office  of  the  Assistant  Secretary  of  Defense/Health  Affairs  tasked 
the  Agency  for  Healthcare  Research  and  Quality  (AHRQ)  to  independently  evaluate  the  best 
practices  in  Crew  Resource  Management  (CRM)-derived  training  programs  for  military  medical 
settings.  The  American  Institutes  for  Research  (AIR)  was  awarded  a  contract  by  AHRQ  to 
conduct  this  research. 

This  report  describes  the  results  of  one  of  the  major  tasks  performed  under  this  effort,  an 
independent  case  study  analysis  of  Department  of  Defense  (DoD)-sponsored  medical  team 
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training  (MTT)  programs.  Programs  reviewed  include  MedTeams  ,  Medical  Team 
Management  (MTM),  and  Dynamics  Outcomes  Management® (DOM).*  The  results  described 
herein  provide  unbiased  infomiation  about  each  MTT  program.  Prior  to  this  report,  program 
developers  had  investigated  their  own  MTT  program’s  effectiveness.  Therefore  this  investigation 
was  the  first  independent  assessment.  However,  although  this  investigation  was  an  independent 
assessment,  the  investigators  only  had  limited  access  to  certain  kinds  of  information.  Thus,  they 
refrained  from  making  judgments  about  the  “goodness”  or  “badness”  of  each  program.  Rather 
the  investigators  provide  comprehensive  descriptive  information. 

Approach 

Several  sources  of  information  were  used  when  gathering  data  on  MedTeams  ,  MTM,  and 
DOM.  First,  AIR  investigators  reviewed  student  and  instructor  guides,  slides,  and  other  audio¬ 
visual  materials  that  course  developers  provided  to  them.  Document  review  also  included 
reviewing  the  relevant  research  studies  that  have  been  published  on  a  specific  program’s 
effectiveness.  Second,  AIR  staff  attended  and  observed  the  classroom  portion  of  each  of  the 
courses.  Furthermore,  AIR  collected  pretraining  data  on  student  experiences  and  expectations 
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and  new,  independent  posttraining  data  on  student  reactions  to  MedTeams  ,  MTM,  and  DOM. 
Finally,  for  MedTeams  and  MTM,  trained  AIR  staff  conducted  one-on-one  interviews  with 
instructors  who  had  taught  or  were  going  to  teach  the  course. 

Results 

The  results  suggested  MedTeams  ,  MTM,  and  DOM  possess  several  desirable 
characteristics,  such  as  using  active  learning  techniques  to  develop  the  participant  teamwork- 
related  competencies  and  offering  interdisciplinary  training  to  teams  of  physicians,  nurses, 
technicians,  and  other  heath  care  professionals.  Nevertheless,  each  program  also  had  a  number  of 
limitations.  For  example,  none  of  the  programs  were  based  on  a  comprehensive  pretraining  needs 
analysis;  limited  opportunities  existed  for  participants  to  receive  structured  practice  and  feedback 
on  critical  teamwork  skills;  and  few  strategies  were  available  for  sustaining  and  reinforcing 
teamwork  principles  in  the  posttraining  environment. 


*  Dynamic  Outcomes  Management  has  been  renamed  LifeWings^'^. 
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Conclusions 

Based  on  the  findings  from  this  investigation  and  other  tasks  performed  under  this  contract, 
AIR  defines  a  successful  medical  team  training  program  as  including  the  following 
characteristics: 

•  A  comprehensive  pretraining  needs  analysis  that  identifies  an  organization’s  readiness  for 
change,  possible  barriers  to  training  transfer,  and  common  and  unique  team  requirements 
as  a  function  of  medical  specialty  area. 

•  A  set  of  validated  team  knowledge,  skill,  and  attitude  competencies  that  were  identified 
as  important  during  the  needs  analysis. 

•  Awareness  training  that  highlights  the  importance  of  human  factors  principles  in 
medicine  and  develops  important  team-related  knowledge  competencies. 

•  Multiple  opportunities  for  trainees  to  receive  structured  practice  and  feedback  on  specific 
team  skills  through  simulation,  on-the-job  training,  or  some  combination  of  the  two. 

•  Recurrent  training  that  reinforces  team  knowledge  and  skills  to  prevent  skill  decay  over 
time. 

Recommendations 

In  addition  to  defining  the  key  features  of  a  successful  medical  team  training  program,  the 
investigators  made  a  number  of  recommendations  to  advance  both  the  practice  and  science  of 
MTT.  These  recommendations  are  as  follows. 

•  First,  they  recommend  that  the  DoD  develop  a  standard  list  of  generic  teamwork-related 
knowledge,  skill,  and  attitude  competencies  that  represent  the  core  elements  of  successful 
team\vork  in  health  care. 

•  Second,  the  investigators  recommend  that  DoD  and  AHRQ  identify  how  team 
knowledge,  skill,  and  attitude  competency  requirements  vary  by  medical  practice,  acuity 
or  other  job  characteristics.  Identification  of  such  competencies  would  lead  to  practice- 
specific  team  requirements  and  tailored  MTT  for  specific  disciplines. 

•  Third,  they  recommend  that  instructional  designers  look  beyond  aviation  CRM  training 
and  leverage  all  available  research  and  tools  (i.e.,  a  tremendous  amount  of  research  on 
teams  has  been  conducted  by  the  U.S.  Navy)  when  developing  medical  team  training 
programs. 

•  Fourth,  the  investigators  recommend  that  the  DoD  develop  a  detailed  training 
specification  for  MTT  programs  for  awareness,  practice  and  feedback,  and  recurrent 
training.  In  addition  to  describing  the  core  teamwork  competencies,  the  specification 
would  outline  the  appropriate  instructional  strategies  for  each  core  competency,  the 
appropriate  sequencing  of  training  activities,  and  outcome  measures  for  assessing  the 
degree  of  skill  acquisition. 
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•  Finally,  they  recommend  that  AHRQ  develop  technical  assistance  documents  under  the 
Patient  Safety  and  Quality  Improvement  Act  of  2005  (P.L.  109-41)  on  issues  related  to 
team  training  and  error  prevention,  much  like  the  FAA’s  advisory  circulars.  The 
investigators  believe  that  human  factors-related  advisory  circulars  would  go  a  long  way 
to  educate  the  medical  community  about  the  importance  of  MTT  for  ensuring  patient 
safety  and  for  ensuring  consistency  across  MTT  programs. 

Summary 

In  conclusion,  this  report  presents  an  in-depth  case  study  analysis  of  MedTeams  ,  Medical 
Team  Management,  and  Dynamic  Outcomes  Management®.  This  was  the  first  independent 
assessment  of  these  programs.  The  case  study  approach  allowed  for  the  collection  of  detailed, 
comprehensive  information  on  each  program,  which  was  reported  along  a  common  set  of 
variables. 
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Chapter  1.  Introduction  and  Methodology* 

Background 

Patient  safety  is  a  top  priority  in  health  care.  The  Institute  of  Medicine’s  (lOM)  publication. 
To  Err  is  Human:  Building  a  Safer  Health  System,^  concluded  that  medical  errors  cause  up  to 
98,000  deaths  annually.  The  lOM  report  brought  national  focus  to  this  important  issue,  and  has 
now  spawned  significant  research  on  the  underlying  causes  of  medical  errors  and  the 
effectiveness  of  different  strategies  for  improving  patient  safety.  Although  still  in  its  infancy, 
research  on  strategies  to  improve  patient  safety  appears  to  have  a  bright  future. 

The  Quality  Interagency  Coordination  Task  Force  (QuIC)  was  established  in  1998  to  address 
a  number  of  critical  needs  identified  in  the  lOM  report.  The  QuIC  is  composed  of  representatives 
from  different  Federal  agencies  including  the  Departments  of  Health  and  Human  Services 
(HHS),  Labor  (DoL),  Defense  (DoD),  and  Veterans  Affairs  (VA),  to  name  a  few.  Currently,  the 
QuIC  has  completed  work  on  existing  projects  and  is  awaiting  input  from  members  on  future 
initiatives  to  pursue.  The  QuIC  identified  100  activities  for  improving  patient  safety,  most  of 
which  have  been  implemented  by  Federal  health  care  organizations.  Among  the  QuIC’s 
recommendations  is  the  widespread  adoption  of  human  factors-based  training,  such  as  Crew 
Resource  Management  (CRM)  training,  for  improving  teamwork  in  health  care.  The  QuIC 
believes  that  lessons  learned  from  high-risk  environments  should  be  looked  at  when  developing 
new  patient  safety  practices. 

Helmreich  and  Foushee  concluded  that  the  introduction  of  CRM  has  been  one  of  the  greatest 
success  stories  in  aviation.  The  efficacy  of  CRM  has  been  established,  in  part,  because  CRM  has 
been  evaluated  throughout  its  evolution.  CRM  training  was  developed  interactively — introducing 
and  testing  the  effectiveness  of  different  strategies,  which  allowed  for  the  best  possible  results.^ 
CRM  training  is  the  gold  standard  for  what  can  be  produced  when  those  interested  in  research 
and  those  interested  in  practice  work  cooperatively  to  achieve  a  common  goal. 

The  Federal  Aviation  Administration  (FAA)  has  issued  specific  guidance  on  the  development 
and  conduct  of  CRM  training.^^  The  FAA  describes  CRM  in  three  phases:  an  awareness  phase,  a 
practice  and  feedback  phase,  and  a  continual  reinforcement  phase.  The  first  stage  of  CRM 
training,  awareness  training,  involves  communicating  teamwork  principles  and  concepts  that  are 
fundamental  to  a  particular  task  domain  and  developing  attitudes  and  beliefs  that  will  motivate 
trainees  to  be  receptive  to  those  ideas."^’^  The  second  stage  of  CRM  training,  skills  practice  and 
feedback,  involves  developing  the  skills  necessary  to  apply  the  concepts  that  were  introduced  in 
the  awareness  stage  to  on-the-job  situations.  This  stage  is  critical  because  “individuals  may 
accept,  in  principle,  abstract  ideas  of  [crew  resource  management  concepts]  but  may  find  it 
difficult  to  translate  them  into  behavior”  on-the-job.^  The  final  stage  of  CRM  training,  continual 
reinforcement,  involves  repeated  exposure  to  CRM  concepts,  as  well  as  on-the-job  feedback  and 
reinforcement  of  CRM  concepts  from  multiple  sources. 

Perceived  parallels  between  health  care  and  aviation  have  led  to  a  number  of  CRM-derived 
medical  team  training  programs.  Applications  of  CRM  in  medicine  started  with  the  introduction 
of  Anesthesia  Crisis  Resource  Management  (ACRM)  at  Stanford  University  School  of  Medicine 


*  Note:  Appendixes  cited  in  this  report  are  provided  electronically  at 
http://www.ahrq ■aov/qu3l/teamtrain/index.html#app . 

^  The  current  version  is  Advisory  Circular  AC  120-5 IE,  dated  Jan.  22,  2004. 
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and  the  Anesthesiology  Service  at  the  Palo  Alto  (CA)  Veteran  Affairs  Medical  Center.^  More 
recently,  DoD  has  funded  several  medical  team  training  initiatives.  MedTeams  has  been 
implemented  in  Army  and  Navy  hospitals,  while  Medical  Team  Management  (MTM)  has  been 
introduced  in  the  U.S.  Air  Force.^  However,  these  training  programs  have  not  undergone  the 
same  scrutiny  as  CRM.  Some  evaluation  studies  have  been  conducted  on  ACRM,  MedTeams™, 
and  MTM,  but  not  with  sufficient  rigor  to  draw  firm  conclusions  about  each  program’s 
effectiveness  or  the  relative  effectiveness  of  different  training  strategies  for  promoting  teamwork 
in  health  care  and  reducing  negative  patient  outcomes. 

The  importance  of  training  evaluation  is  unquestioned  by  instructional  designers  and  training 
researchers;  however,  training  is  often  designed  and  developed  but  not  evaluated.  Training 
evaluation  has  been  defined  as  “the  systematic  collection  of  descriptive  and  judgmental 
information  necessary  to  make  effective  decisions  related  to  . . .  various  instructional  activities.”^ 
These  decisions  include  determining  whether  the  goals  and  objectives  of  a  program  are 
appropriate  to  achieve  the  desired  outcome,  whether  the  content  and  methods  in  training  will 
result  in  achievement  of  the  overall  program  goals,  and  how  to  maximize  training  transfer. 
Although  systematic  training  evaluation  is  not  an  easy  task,  it  is  the  only  way  to  ensure  that 
training  programs  have  the  desired  effect  and  are  a  worthwhile  investment  for  an  organization. 

Purpose  of  this  Investigation 

The  purpose  of  this  investigation  is  to  conduct  a  case  study  review  of  DoD-sponsored 
medical  team  training  (MTT)  programs.  The  American  Institutes  of  Research  (Washington,  DC) 
conducted  the  investigation  under  contract  to  AHRQ  during  the  summer  of  2003.  Programs 

TT'Xyf 

reviewed  include  MedTeams  ,  Medical  Team  Management  (MTM),  and  Dynamics  Outcomes 
Management®  (DOM).^ 

A  case  study  approach  was  selected  for  this  initial  investigation  because  we  did  not  have  the 
access  or  the  resources  to  conduct  an  empirically  sound,  comparative  analysis.  For  example, 
convenience  samples  were  relied  upon  when  collecting  data  from  program  participants.  This 

*T*Xyf 

resulted  in  data  being  collected  fi-om  MedTeams  participants  from  a  DoD  facility,  future  MTM 
instructors,  and  DOM  participants  fi*om  a  university  medical  center.  Moreover,  we  only  had 
access  to  the  classroom  (awareness  phase)  portion  of  each  training  program.  All  of  the  programs 
address  the  need  for  skills  practice  and  feedback  through  a  variety  of  post-classroom  activities, 
however,  due  to  time  constraints  associated  with  this  contract,  we  did  not  have  an  opportunity  to 
observe  these  actions.  Therefore,  it  is  important  to  keep  in  mind  several  key  characteristics  of 
this  investigation  when  reviewing  this  report.  First,  this  report  describes  case  studies  of  three 
existing  medical  team  training  programs:  MedTeams  ,  MTM,  and  DOM.  Although  data  for 
each  case  study  were  described  along  the  same  set  of  dimensions,  this  information  is  not 
statistically  comparable  because  it  was  derived  from  different  information  sources  with  different 
levels  of  rigor.  Second,  the  goal  of  this  report  was  to  provide  unbiased  information  to  the  DoD 
and  AHRQ  about  these  programs  as  a  whole.  The  program  developers  have  conducted  past 
investigations,  and  therefore  this  investigation  was  the  first  independent  assessment.  Finally, 
because  we  only  had  access  to  certain  kinds  of  information,  we  refrain  from  making  judgments 
about  the  “goodness”  or  “badness”  of  each  program.  Rather,  we  provide  comprehensive 
descriptive  information. 


^  Dynamic  Outcomes  Management  has  been  renamed  LifeWings™. 
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Prior  to  describing  each  program,  we  first  present  an  overview  of  training  evaluation.  Here, 
we  will  describe  the  prevailing  approaches  for  determining  training  effectiveness,  both  within 
and  outside  of  health  care.  Different  approaches  exist  and  we  will  review  the  relative  merits  of 
each.  Second,  we  present  our  case  study  approach  for  MedTeams™,  MTM,  and  DOM.  To  the 
extent  possible  we  relied  upon  best  practices  from  the  literature  to  develop  our  approach. 
However,  we  had  to  adapt  many  of  these  practices  to  the  unique  characteristics  of  the  current 
situation.  In  this  study,  we  were  highly  constrained  by  the  extent  to  which  we  could  gain  access 
to  the  participants  who  completed  each  training  program.  Our  case  study  approach  to 
investigating  these  programs  emerged  from  these  constraints.  Finally,  we  describe  the 
individuals  who  participated  in  MedTeams™,  MTM,  and  DOM.  For  each  participant  we 
collected  information  on  several  critical  pretraining  factors  that  can  influence  motivation  to  learn 
during  training. 


Training  Evaiuation 

Training  evaluation  is  perhaps  one  of  the  most  difficult,  yet  most  important,  activities  in  the 
instructional  development  process.  The  purpose  of  training  evaluation  is  to  systematically  collect 
data  about  a  training  program  to  determine  the  program’s  overall  effectiveness.^  Evaluation  can 
be  used  to  determine  whether  people  liked  the  training  program,  whether  they  learned  what  was 
intended  for  them  to  learn,  and  whether  it  positively  affected  their  job  performance.^^  It  can  also 
be  used  to  gather  information  about  specific  aspects  of  a  training  strategy  such  as  the  usefulness 
of  training  materials  (e.g.,  manuals  and  videotapes),  attitudes  about  new  technologies  (e.g., 
automation)  and  instructional  techniques  (e.g.,  lecture  versus  practice  and  feedback).  When 
multiple  training  programs  are  evaluated  on  the  same  dimensions,  training  programs  can  be 
compared  to  one  another,  as  can  different  groups  of  trainees. 

A  number  of  approaches  have  been  proposed  for  conducting  detailed  training  evaluation 
within  different  disciplines.  Below  we  review  evaluation  strategies  from  the  domains  of 
instructional  systems  design  and  health  care.  Then  we  describe  the  approach  that  guided  our  case 
studies  of  MedTeams™,  MTM,  and  DOM. 

Instructional  Systems  Design 

Kirkpatrick^^  proposed  one  of  the  first  and  still  most  influential  models  of  training  evaluation 
in  which  he  outlined  four  types  (Level  I,  Level  II,  Level  III,  and  Level  IV)  of  outcome  measures: 
reactions,  learning,  behavior,  and  results. 

•  Reactions  (Level  I)  are  simply  measures  of  how  well  trainees  liked  or  valued  a  training 
program  and  certain  aspects  of  the  program. 

•  Learning  (Level  II)  involves  measuring  the  extent  to  which  trainees  understand  and  retain 
principles,  facts,  and  techniques  that  are  imparted  during  training. 

•  Behavior  (Level  III)  refers  to  measures  of  any  behavioral  changes  that  occur  as  a  result  of 
training. 

•  Results  (Level  IV)  refer  to  the  measurement  of  the  impact  of  training  on  organizational 
criteria;  criteria  such  as  increased  profitability  or  enhanced  customer  satisfaction  would 
be  indications  of  training  effectiveness. 

A  review  of  the  training  literature  suggests  that  Kirkpatrick’s  Level  I  data,  trainee  reactions, 
are  the  most  commonly  collected  outcome  when  evaluating  training.  Reaction  measures  focus 
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1  ^ 

on  a  trainee’s  affective  reactions  to  training,  under  the  assumption  that  the  extent  to  which 
trainees  liked  the  training  is  correlated  with  performance  on  the  job.  Research  that  has  explored 
the  relationship  between  trainee  reactions  and  other  training  outcomes,  however,  has  found  that 
trainee  affective  reactions  correlate  only  slightly  with  learning  and  on-the-job  performance  (i.e., 
behavior).  ’  Thus,  simply  asking  trainees  whether  they  like  a  training  program  does  not 
necessarily  predict  whether  they  actually  learn  anything  or  alter  their  behavior  on  the  job. 
Nonetheless,  affective  reactions  are  important.  Trainees  are  the  customers  of  training  and 
negative  reactions  to  training  can  undermine  the  credibility  of  training  and  the  sponsoring 
organization. 

Recent  research  has  indicated  that  asking  trainees  about  the  instrumentality  or  utility  of 

training  is  preferred  to  simply  asking  them  whether  or  not  they  liked  the  training.  An  example  of 

this  type  of  question  might  be  to  ask  trainees  to  indicate  the  extent  to  which  they  perceived 

training  to  be  of  practical  value.  Such  questions  seek  to  determine  the  usefulness  of  training  for 

1  ^  1/1 

performing  a  job.  Alliger  and  his  colleagues  ’  found  that,  although  affective  reactions  to 
training  (e.g.,  “Training  was  enjoyable.”)  did  not  relate  to  subsequent  job  performance,  utility 
reactions  (e.g.,  “Training  was  useful.”)  were  related  to  both  learning  and  performance  on  the  job. 
Interestingly,  these  researchers  found  that  measures  of  learning  correlated  less  with  on-the-job 
performance  than  either  utility  reactions  or  the  combination  of  affective  and  utility  reactions. 

In  addition  to  the  recent  advancements  made  by  Alliger  and  his  colleagues,  other  researchers 
have  made  contributions  to  the  domain  of  training  evaluation  by  expanding  the  Kirkpatrick 
model.  Kraiger  and  his  colleagues  advocated  a  multidimensional  view  of  Kirkpatrick’s  Level  II 
data.  These  researchers  decomposed  learning  into  cognitive,  affective,  and  skill-based  outcomes. 
Similarly,  Kraiger  and  Jung^^  suggested  several  processes  by  which  learning  outcomes  can  be 
derived  from  and  linked  to  instructional  objectives  of  training.  Finally,  Baldwin  and  Magjuka^^ 
pointed  out  that  outcomes  are  influenced  by  a  number  of  antecedent  conditions  that  exist  and 
interact  within  an  organization.  These  include  the  manner  in  which  training  is  introduced 
(voluntary/mandatory,  goals,  etc.),  trainee  past  experience,  pretraining  motivation,  and  the  extent 
to  which  a  positive  transfer  climate  exists  within  the  organization.  For  example,  mandated 
training  will  be  successful  when  employees’  past  experience  with  similar,  mandated  programs 
has  been  positive,  leadership  endorses  the  importance  of  the  training  program,  and  new 
knowledge  and  skills  are  reinforced  on  the  job. 

Analytic  Models  of  Health  Care 

Donabedian’s  structure-process-outcome  model  serves  as  a  unifying  framework  for 
examining  health  services  and  assessing  patient  outcomes. Donabedian  defined  structure  as  the 
physical  and  organizational  properties  of  the  settings  in  which  care  is  provided, as  the 
actual  treatments  and  procedures  that  are  done  for  patients,  and  outcomes  as  what  is  actually 
accomplished  for  patients.  From  the  standpoint  of  patient  safety,  Donabedian’s  model  provides  a 
framework  for  examining  how  risks  and  hazards  that  are  embedded  within  the  structure  and 
process  of  care  have  the  potential  to  cause  injury  or  harm  to  patients.  For  example,  individual  or 
team  failures  on  the  part  of  health  care  providers  have  been  consistently  cited  as  leading  to 
negative  patient  outcomes. 

1  o 

Coyle  and  Battles  modified  the  Donabedian  model  to  include  important  antecedent 
conditions  that  can  affect  patient  outcomes.  These  researchers  suggested  that  accounting  for 
environmental  and  patient  factors  is  critical  in  understanding  the  effectiveness  of  any  new 
strategy  that  is  introduced  or  modifications  that  are  made  to  the  patient  care  process.  In  health 
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care,  improving  patient  outcomes  is  the  ultimate  criterion  for  a  strategy  to  be  deemed  successful. 
A  change  in  process  must  lead  to  a  corresponding  change  in  patient  outcomes.  Under  patient 
factors,  Coyle  and  Battles  included  genetics,  sociodemographics,  health  habits,  beliefs  and 
attitudes,  and  preferences;  under  environmental  factors  they  included  cultural,  social,  political, 
personal,  physical,  and  other  factors  related  to  the  health  professions. 

Case  Study  Approach 

For  each  training  program,  we  were  able  to  access  several  sources  of  information.  First,  we 
reviewed  all  available  documentation.  Document  review  included  reviewing  all  student  and 
instructor  guides,  slides,  and  other  audio-visual  materials  that  course  developers  provided  to 
AIR.  Document  review  also  included  reviewing  the  relevant  research  studies  that  have  been 
published  on  a  specific  program’s  effectiveness.  In  addition  to  our  document  review,  we 
observed  the  classroom  portion  of  each  of  the  courses.  Furthermore,  for  each  course  we 
observed,  we  collected  pretraining  data  on  student  experiences  and  expectations  and  new, 
independent  posttraining  data  on  student  reactions  to  MedTeams  ,  MTM,  and  DOM.  Finally, 
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for  MedTeams  and  MTM,  we  conducted  one-on-one  interviews  with  instmctors  who  had 
taught  or  were  going  to  teach  the  course.  Each  of  these  activities  is  briefly  described  in  more 
detail  below. 

Document  and  Literature  Review 

As  part  of  a  larger  state-of-the  art  literature  review,  AIR  reviewed  the  available  research  on 
medical  team  training  and  specific  medical  team  training  programs.  This  review  identified  eight 
articles,  conference  papers,  or  technical  reports  published  on  the  MedTeams  approach  and  one 
article  describing  the  effectiveness  of  DOM.  Key  to  our  assessment  of  MedTeams  was  an 
article  published  by  Morey  and  colleagues^  ^  that  describes  an  evaluation  of  the  Emergency 
Department  (ED)  curriculum.  Key  to  our  assessment  of  DOM  was  an  article  published  by  Rivers 
and  colleagues^^  that  describes  an  evaluation  of  DOM  training.  No  comparable  investigations 
were  found  for  MTM.  Results  of  these  papers  are  described  when  we  review  additional  evidence 
that  supports  each  medical  team  training  curriculum. 

In  addition  to  reviewing  the  literature,  we  carefully  reviewed  the  instructor  and  student 
guides.  For  MedTeams™,  we  reviewed  the  instmctor’s  guide  for  the  Labor  &  Delivery  (L&D) 
Team  Coordination  Course  and  the  student’s  guide  for  the  Operating  Room  (OR)  Team 
Coordination  Course.  For  DOM,  we  reviewed  the  student’s  guide  for  a  course  conducted  at  a 
university  medical  center.  Finally,  we  reviewed  the  instmctor’s  guide  for  MTM.  Course 
developers  provided  these  guides  to  us  when  we  attended  and  observed  each  class. 

Course  Observations  and  Outcome  Reviews 

Course  observations.  AIR  staff  observed  at  least  one  session  of  the  classroom  portion  of 
each  of  the  three  training  curricula.  For  MedTeams  ,  we  observed  the  train-the-trainer  portion 
of  the  L&D  Team  Coordination  Course  during  fall  2002  in  Boston,  MA,  and  1 1  sessions  of  the 
OR  Team  Coordination  Course  at  a  U.S.  naval  medical  center  during  summer  2003.  For  MTM, 
we  observed  the  train-the-trainer  course  that  was  conducted  during  summer  2003  at  a  U.S.  Air 
Force  base.  Finally,  for  DOM,  we  observed  a  shortened  version  of  the  curriculum  that  was 
sponsored  by  the  State  Volunteer  Mutual  Insurance  Company  (Brentwood,  TN)  and  two  full- 
length  sessions  of  the  course  that  were  conducted  at  a  university’s  regional  medical  center. 
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Pre-  and  posttraining  assessment  tools.  For  each  of  the  courses  that  were  observed  (except 
the  MedTeams™  L&D  Team  Coordination  Course  and  the  shortened  version  of  DOM),  AIR 
staff  administered  pre-  and  posttraining  assessment  tools.  The  pretraining  tool  was  designed  to 
assess  student  experiences  prior  to  training  and  gauge  participant  attitudes  and  beliefs  about 
teamwork  (see  Appendix  A).  Pretraining  experiences  and  participant  attitudes  have  been  shown 
to  affect  trainee  motivation  to  learn  and  reactions  to  training.  ’  The  posttraining  tool  measured 
participant  affective  and  utility  reactions  to  training  (see  Appendix  B). 

A 

Instructor  interviews.  During  our  observations  of  MedTeams  and  MTM,  we  also 

conducted  instmctor  interviews.  With  respect  to  MedTeams  ,  we  interviewed  14  hospital  staff 

TTVl 

members  during  the  week  of  June  9-11, 2003,  who  served  as  instructors  in  a  MedTeams 
program  that  week.  With  respect  to  MTM,  we  interviewed  10  individuals  who  were  attending  or 
had  previously  attended  a  train-the-trainer  course.  In  all  cases,  instructors  were  asked  about  how 
they  were  selected,  their  background  in  training,  and  perceptions  of  the  course  and  training  aids 
used  (see  Appendix  C).  Interviews  were  not  conducted  with  the  two  DOM  instructors,  because, 
unlike  MedTeams  and  MTM,  DOM  does  not  utilize  a  train-the-trainer  strategy.  The  course 
developer.  Crew  Training  International  (CTI),  provides  instructors  for  all  administration  of  DOM 
training. 

Description  of  the  Participants 

AIR  staff  interviewed  14  instructors,  and  collected  pre-  and  posttraining  assessments  on  223 

A 

MedTeams  participants;  interviewed  10  participants  and  former  participants,  and  collected 
pre-  and  posttraining  assessments  on  26  MTM  participants;  and  collected  78  pre-  and  post¬ 
training  assessments  from  DOM  participants.  Below  we  describe  our  findings  from  the  pre¬ 
training  assessment,  which  were  used  to  determine  if  participants  were  similar  across 
MedTeams™,  MTM,  and  DOM  training. 

The  pretraining  results  highlight  some  interesting  similarities  among  the  three  groups  of 
respondents  (see  Table  1).  We  began  by  reviewing  their  motivation  to  participate  in  medical 
team  training.  With  the  exception  of  one  question  (Question  #3),  nearly  50  percent  or  more  of  all 
participants  had  personally  witnessed  a  breakdown  in  teamwork  that  could  have  compromised 
patient  safety.  In  all  cases,  the  two  most  frequently  cited  problems  involved  not  learning  from 
prior  mistakes  (Question  #4),  and  poor  preparation  (Question  #5).  Fortunately,  the  participants 
did  not  report  feeling  pressured  to  perform  procedures  that  they  were  not  comfortable  doing 
(Question  #3).  To  empirically  assess  the  similarity  across  the  MedTeams™,  MTM,  and  DOM 
participants,  we  ranked  the  percentage  of  agreement  responses  (within  each  program)  and 
calculated  the  mean  correlation  across  the  three  programs,  using  Spearman’s  correlation  for 
ranked  data  (rg).  The  average  correlation  was  0.69,  indicating  a  moderate  degree  of  similarity 
among  participants. 

We  found  similar  results  with  regard  to  the  participants’  respective  organizational  cultures.  In 
all  cases,  over  50  percent  of  the  participants  in  each  program  agreed  that  their  respective 
organizations  promoted  a  positive  safety  culture  (see  Table  2).  Participants  in  all  three  programs 
generally  agreed  that  disruptions  in  patient  care  were  the  greatest  detriments  to  patient  safety 
(Question  #8).  However,  they  were  uniformly  less  likely  to  agree  that  team  members  in  their 
departments  know  each  others’  responsibilities  (Question  #9).  Again,  we  calculated  the  mean 
correlation  across  the  three  programs,  using  Spearman’s  correlation  for  ranked  data  (rs).  The 
average  correlation  was  0.46,  which  is  somewhat  lower  than  their  pretraining  motivation  levels. 
This  was  not  unexpected,  given  that  each  facility  is  expected  to  have  its  own  unique  culture. 
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Table  1.  Pretraining  motivation  (%  “yes”  responses) 


Survey  item 

MedTeams™ 

(n=223) 

MTM 

{n=26) 

DOM 

(n=77) 

1.  Have  you  ever  worked  in  a  medical  team  where  you 
did  not  feel  comfortable  voicing  your  professional 
opinion? 

46% 

58% 

68% 

2.  Have  you  ever  worked  in  a  medical  team  where 
there  was  no  clearly  designated  leader? 

48% 

46% 

64% 

3.  Have  you  ever  felt  pressured  to  perform  a  medical 
procedure  that  you  felt  uncomfortable  doing? 

33% 

46% 

56% 

4.  Have  you  ever  witnessed  an  actual  or  potential 
mishap  that  could  have  been  prevented  if  the  team 
had  learned  from  their  previous  mistakes? 

58% 

63% 

86% 

5.  Have  you  ever  witnessed  a  routine  medical 
procedure  that  went  wrong  because  the  team  did  not 
adequately  prepare  beforehand? 

60% 

50% 

77% 

Finally,  we  reviewed  the  participants’  belief  in  the  importance  of  teamwork.  The  participants 
uniformly  agreed  that  teamwork  was  important  for  ensuring  patient  safety.  In  fact,  the  percentage 
agreement  for  these  items  was  generally  greater  than  80  percent  and  never  went  below  73 
percent  (see  Table  3).  Prior  to  participating  in  their  respective  MTT  programs,  the  participants 
were  uniformly  confident  in  their  ability  to  work  effectively  in  a  team  environment  (Question 
#12)  and  in  their  belief  that  people  with  strong  teamwork  skills  are  likely  to  be  successful  in 
health  care  (question  #14).  Again,  we  calculated  the  mean  correlation  among  the  three  programs 
using  Spearman’s  correlation  for  ranked  data  (rs).  The  average  correlation  was  .73,  indicating  a 
moderate  degree  of  similarity  among  participants. 

In  summary,  although  the  results  did  differ  slightly,  there  was  a  common  pattern  of  responses 
across  MedTeams™,  MTM,  and  DOM  participants.  Regardless  of  the  institution  in  which 
training  was  delivered,  nearly  half  of  all  participants  had  witnessed  a  breakdown  in  teamwork 
that  could  have  compromised  patient  safety  (see  Table  1).  Despite  this,  many  felt  that  their 
organizations  exhibited  elements  of  positive  safety  culture  (see  Table  2),  and  that  teamwork 
skills  are  important  for  maintaining  patient  safety  (see  Table  3).  Therefore,  we  conclude  that 
participants  in  MedTeams™,  MTM,  and  DOM  possessed  similar  motivation  to  participate  in 
training  and  learn  the  material.  With  that  in  mind,  we  present  our  case  studies  of  MedTeams™, 
MTM,  and  DOM. 
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Table  2.  Organizational  culture  (%  agreement) 


Survey  Item 

MedTeams™ 

(n=223) 

MTM 

(n=26) 

DOM 

(n=77) 

6.  The  culture  in  our  department  makes  it  easy  to 
learn  from  the  mistakes  of  others. 

68% 

81% 

51% 

7.  Our  doctors,  nurses,  enlisted  personnel,  and  other 
team  members  work  together  as  a  well-coordinated 
team. 

57% 

52% 

63% 

8.  Disruptions  in  patient  care  can  be  detrimental  to 
patient  safety. 

76% 

88% 

69% 

9.  Physicians,  nurses,  enlisted  personnel,  and  other 
team  members  in  this  department  know  and 
understand  each  others’  respective  responsibilities. 

53% 

52% 

54% 

10.  My  department  does  a  good  job  of  training  new 
personnel. 

53% 

58% 

51% 

Table  3.  Belief  in  the  importance  of  teamwork  (%  agreement) 


Survey  Item 

MedTeams^^ 

MTM 

DOM 

(n=223) 

(n=26) 

(n=77) 

1 1 .  Teamwork  deserves  more  attention  In  health  care. 

81% 

92% 

97% 

12.  lam  confident  about  my  ability  to  work  effectively 
in  a  team. 

97% 

96% 

96% 

13.  Teamwork  is  one  of  the  most  important  skills  in 
the  operating  room  (OR). 

92% 

92% 

87% 

14.  People  with  strong  teamwork  skills  are  more  likely 
to  be  successful  in  health  care. 

94% 

100% 

99% 

15.  It  is  impossible  to  function  in  health  care  without 
being  a  good  team  player. 

73% 

77% 

81% 
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Chapter  2.  Case  Study  1 — MedTeams™* 

Introduction 

The  primary  purpose  of  MedTeams  is  to  reduce  medical  errors  through  interdisciplinary 
teamwork.  MedTeams™  was  developed  by  Dynamics  Research  Corporation  (DRC),  of 
Andover,  MA,  on  the  premise  that  most  errors  result  from  breakdowns  in  systems-level  defenses 
that  occur  over  time.  According  to  the  MedTeams  curriculum,  each  team  member  has  a 
vested  interest  in  maintaining  patient  safety  and  is  expected  to  take  an  assertive  role  in  breaking 
the  chain  of  events  leading  to  an  error.  MedTeams™  defines  a  core  team  as  a  group  of  3-10 
(average  =  6)  medical  personnel  who  work  interdependently  during  a  shift  and  who  have  been 
trained  to  use  specific  teamwork  behaviors  to  coordinate  their  clinical  interactions. 

Case  Study  Approach 

Consistent  with  our  plan,  we  were  able  to  access  several  sources  of  information  to  evaluate 
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MedTeams  training.  Specially,  we  reviewed  all  course  materials  for  both  the  MedTeams 
Instructor  Certification  Course  for  Labor  &  Delivery  (L&D)  and  the  student  guides  for  the 
Operating  Room  (OR)  Course.  We  also  reviewed  the  relevant  research  studies  that  have  been 
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published  on  the  effectiveness  of  MedTeams  training.  In  addition  to  our  document  review,  we 
observed  1  day  of  the  L&D  train-the-trainer  course  and  multiple  sessions  of  the  OR  student 
course.  At  the  OR  course,  we  also  were  able  to  conduct  a  pretraining  assessment  of  student 
experiences  and  expectations,  a  posttraining  assessment  of  student  reactions  to  MedTeams  , 
and  interviews  with  14  instructors.  Each  of  these  activities  is  briefly  described  in  more  detail 
below. 

Document  and  Literature  Review 

Our  document  review  identified  eight  articles,  conference  papers  or  technical  reports 
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published  on  the  MedTeams  approach.  Key  to  our  assessment  of  MedTeams  training  was  a 

paper  published  by  Morey  and  colleagues^^  describing  a  quasi-experimental  evaluation  of  the 
effectiveness  of  this  training.  Results  from  this  paper  are  described  when  we  review  the 
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additional  evidence  that  supports  the  MedTeams  approach. 

In  addition  to  reviewing  the  literature,  we  carefully  reviewed  the  instructor  and  student 
guides  from  the  L&D  Team  Coordination  Course  and  the  OR  Team  Coordination  Course, 
respectively.  DRC  staff  provided  these  guides  to  us  when  we  attended  and  observed  each  of 
these  curriculums. 

Course  Observations 

AIR  staff  observed  the  train-the-trainer  portion  of  the  L&D  Team  Coordination  Course 
during  fall  2002  in  Boston,  MA.  This  course  was  taught  by  DRC  as  part  of  the  large,  on-going 
course  validation  study.  A  variety  of  staff  attended  this  training  from  hospitals  randomly 
assigned  to  the  experimental  condition  for  the  study.  AIR  staff  observed  1  day  of  a  multiday 
course,  which  is  designed  to  certify  hospital  staff  as  MedTeams™  instructors  (as  well  as  meet 
the  requirements  for  participation  in  the  L&D  study).  The  observed  day  involved  DRC 


*  Note:  Appendixes  cited  in  this  report  are  provided  electronically  at 

http://www.ahrq. eov/qual/teamtrain/index.htinl#app  . 
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instructors  providing  the  MedTeams  classroom  training  to  the  future  instructors.  Essentially, 
the  purpose  of  this  day  was  to  demonstrate  how  the  classroom  portion  of  MedTeams™  was  to  be 
conducted.  On  a  subsequent  day,  the  instructor  trainees  would  practice  delivering  this  training. 
This  session  was  not  observed. 

In  addition  to  the  L&D  course,  AIR  staff  observed  the  OR  Team  Coordination  Course  at  a 
U.S.  naval  medical  center  during  summer  2003.  Twelve  classes  were  conducted  over  a  1-week 
period,  including  two  for  the  ophthalmology  service,  one  for  the  dental  service,  three  for  the 
orthopedic  service,  one  for  the  urology  service,  one  for  plastic  surgery,  two  for  general  surgery, 
one  for  the  cardiothoracic  service,  and  one  for  the  neurology  group.  Hospital  staff  who  had 
previously  completed  the  MedTeams  instructor  certification  course  and  had  been  certified  as 
MedTeams  instructors  taught  these  courses.  We  observed  the  didactic  portion  of  this  training. 

Pre-  and  Posttraining  Assessment  Tools 

In  addition  to  observing  the  courses,  AIR  administered  pre-  and  posttraining  assessment  tools 
to  223  participants  in  MedTeams  training.  The  pretraining  assessment  tool  was  designed  to 
assess  student  experiences  prior  to  training  and  gauge  participant  attitudes  and  beliefs  about 
teamwork.  The  results  from  this  measure  were  described  in  an  earlier  section  of  this  report 
entitled,  “Description  of  the  Participants”  (p.  12-14).  The  posttraining  assessment  tool  measured 
participant  utility  reactions  to  MedTeams  training.  All  attendees  completed  the  pretraining 
tool,  but  only  218  completed  the  posttraining  tool.  Of  the  223  students,  94  were  physicians, 
dentists,  or  oral  surgeons;  62  were  corpsmen;  44  were  nurses;  and  23  represented  other  groups, 
such  as  surgical  technicians. 

Instructor  Interviews 

AIR  staff  also  interviewed  14  hospital  staff  who  conducted  the  observed  MedTeams™ 
training.  Interviews  were  conducted  with  trained  instructors  from  the  cardiothoracic,  dentistry, 
general  surgery,  neurology,  ophthalmology,  orthopedics,  plastic  surgery,  and  urology  groups. 

The  pool  of  instructors  included  department  heads,  physicians,  nurses,  and  surgical  technicians. 
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Instructors  were  asked  about  how  they  were  selected  to  teach  MedTeams  ,  their  training 
background,  and  the  flow  of  the  course  and  training  aids  used  (see  Appendix  C  for  the  Instructor 
Interview  Form). 

Below,  we  present  the  results  from  our  analysis  of  this  information.  These  results  are 
organized  around  a  set  of  variables  that  we  used  to  draw  conclusions  about  each  training 
program.  For  each  variable  we  provide  descriptive  data  from  our  literature  review  and  then 
findings  from  our  observations,  data  collections,  and  interviews  when  appropriate.  Table  4 
provides  a  summary  of  this  information  for  each  of  the  three  training  programs  reviewed. 

Results 

Theoretical  Basis 

The  core  objectives,  curriculum,  and  instructional  strategies  for  initial  MedTeams™  training 
for  emergency  departments  (EDs)  were  derived  from  a  CRM  training  program  that  was 
originally  developed  for  U.S.  Army  helicopter  crews  to  train  them  in  specific  behavioral 
skills.^^’^'^  The  MedTeams™  developers  argued  that  emergency  medicine  and  aviation  share  a 
number  of  similarities,  making  CRM  training  for  helicopter  crews  portable  to  the  ED.  These 
similarities  include:  the  need  for  decisionmaking  based  on  incomplete  or  conflicting  information; 
the  demand  for  coordination  among  professionals  with  varied  skills  and  ranks;  and  the  possibility 
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of  poor  team  performance  leading  to  serious  consequences  or  death.  Since  the  development  of 
the  emergency  department  course,  MedTeams™  training  has  been  developed  for  the  L&D 
Service  and  the  OR.  These  courses  are  based  on  the  initial  emergency  department  (ED)  course, 
but  case  studies  and  examples  have  been  modified  to  make  MedTeams™  contextually 
appropriate  for  the  service  in  which  it  is  implemented. 

Analysis  of  Training  Needs 

As  part  of  the  initial  MedTeams  development  process  for  the  ED,  DRC  conducted  an 
analysis  of  team  performance  deficiencies.  They  analyzed  the  closed-case  files  from  eight 
hospitals  that  were  collected  over  a  period  of  several  years.  Each  file  was  then  classified  using  a 
teamwork  failure  checklist  to  identify  trends.  Their  analyses  identified  approximately  8.8 
teamwork  failures  per  closed  case.  Based  on  their  analyses,  DRC  concluded  that  improved 
teamwork  could  have  saved  the  hospitals  approximately  $3.50  per  ED  patient  visit.  By 
way  of  comparison,  malpractice  costs  range  between  $2-$6  per  patient. 

TTlVI 

The  initial  version  of  MedTeams  was  developed  using  an  evaluation-driven  course  design. 
Based  on  the  closed-case  file  review,  DRC  identified  five  critical  teamwork  dimensions  that 
were  necessary  for  effective  teamwork.  They  then  identified  48  specific,  observable  behaviors 
that  were  linked  to  these  dimensions,  and  developed  Behaviorally  Anchored  Rating  Scales 
(BARS)  for  each  behavior.  Finally,  they  reviewed  and  refined  the  curriculum  during  three  5 -day 
expert  panels  that  included  ED  physicians  and  nurses  from  12  hospitals  of  various  sizes.^^’^^ 

Training  Objectives 

The  overarching  objective  of  MedTeams  curriculum  is  to  reduce  medical  errors  through 
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training  interdisciplinary  teamwork  skills.  MedTeams  was  developed  on  the  premise  that  most 
errors  result  from  breakdowns  in  systems-level  defenses  that  occur  over  time.  According  to  the 
MedTeams™  curriculum,  each  team  member  has  a  vested  interest  in  maintaining  patient  safety 
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and  is  expected  to  take  an  assertive  role  in  breaking  the  error  chain.  MedTeams  defines  a  core 
team  as  a  group  of  3-10  (average  =  6)  medical  personnel  who  work  iriterdependently  during  a 
shift,  and  who  have  been  trained  to  use  specific  teamwork  behaviors  to  coordinate  their  clinical 
interactions.  Each  core  team  includes  at  least  one  physician  and  one  nurse.  A  coordinating  team 
manages  several  core  teams,  assigns  new  patients  to  the  core  teams  and  provides  additional 

23,25^6 

resources  as  necessary. 
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Table  4.  Summary  of  medical  team  training  program  content 


Evaluation  Items 

Programs 

MedTeams™ 

MTM 

DOM 

Theoretical  Basis 

>  Based  on  the  CRM* 
training  program 
developed  to  train 

U.S.  Army  helicopter 
crews 

>  Based  on  the  U.S 

Air  Force’s  fighter 
pilot  CRM  training 
program 

>  Based  on  CRM 
training  from 
military  and 
commercial 
aviation 

Needs  Analysis 

>  Analyzed  the 
closed-  case  files 
from  eight  hospitals 

>  Teamwork  and 
communication  as 
root  cause  of 
sentinel  event 

>  No  known  in- 
depth  analysis 

Training 

Objectives 

>  Reduce  medical 
errors  through 
interdisciplinary 
teamwork 

>  Reduce  medical 
errors  by  teaching 
human  factors 
concepts  to 
interdisciplinary 
teams  of  medical 
professionals 

>  Change  the 
military’s  medical 
culture 

>  Apply  aviation 
safety  practices  to 
health  care 

Training  Content 
Knowledge 

Skills 

>  Knowledge  of  the 
components  of 
teamwork 

>  Situational 
awareness 

>  Maintaining  team 
structure  and  climate 

>  Problem-solving 
skills 

>  Execution  of  plans 
and  management  of 
workload 

>  Communication 
skills 

>  Knowledge  of  the 
components  of 
teamwork 

>  Situational 
awareness 

Available  resources 

>  Policy/Regulations 

>  Leadership 

>  Identification  of 
impaired 
performance 

>  Workload 
performance 

>  Communication 
skills 

>  Knowledge  of  the 
components  of 
teamwork 

>  Situational 
awareness 

>  Managing  fatigue 

>  Decisionmaking 

>  Recognizing 
adverse  situations 

>  Cross-check  and 
communication 
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Table  4.  Summary  of  medical  team  training  program  content  (cont.) 


Evaluation  Items 

Programs 

MedTeams™ 

MTM 

DOM 

Skills  (cont.) 

>  Team 
performance 
improvement  skills 
(e.g.,  teamwork 
review,  situational 
learning,  peer 
coaching) 

>  Operating  strategy 

>  Recognition  of 
obstacles  to  effective 
teamwork  and 
communication 

>  Performance 
feedback 

Instructors 

>  Train-the-trainer 

>  MedTeams"^ 
Certified  instructors 

>  Train-the-trainer 

>  Crew  Training 
International 
Instructors 

Instruction 

Strategies 

>  8  hours  of 
classroom 
instruction 

>  1  hour  of 
behavioral 
modeling,  using 
videotaped 
vignettes 

>  1  hour  of 
integration/synthe¬ 
sis 

>  Didactic  lectures 

>  Seminar 
participation 

>  Application 
questions 

>  Behavioral 
modeling 

>  Videotaped 
vignettes 

>  Case  study 
analyses 

>8  hours  of  classroom 
training.  Including: 
lectures,  demon¬ 
strations,  case 
studies,  and  role 
plays 

Practice/Feedback 

>  4  hours  of  on-site 
practice 

>  Routine  team 
meetings  6  months 
after  the  training 

>  Instructors’ 
observations  and 
feedback 

>  Homework 
assignments 

>  Case  studies 

>  Development  and 
Implementation  of 
checklists  and  other 
tools 

Recurrency 

None 

None 

None 

Evaluation 

>  Trainee  reactions 

>  Trainee  reaction 

>  Trainee  reaction 

>  Human  Factors 

Attitude  Survey 

*  CRM  =  Crew  Resource  Management 


Training  Content 

The  MedTeams™  course  consisted  of  a  brief  background  on  teams  and  an  introduction  to 
DoD  Patient  Safety  initiatives.  In  addition,  detailed  information  on  six  substantive  modules  was 
presented.  Each  module  was  structured  around  specific  learning  objectives.  These  objectives 
were  likely  derived  from  the  results  of  the  needs  analysis  that  was  described  earlier  but  we  could 
uncover  no  evidence  to  support  this  hypothesis.  Finally,  modules  did  not  include  any  in-class 
assessments  to  ensure  that  participants  had  achieved  the  stated  learning  objectives.  The  six 
modules  were  as  follows: 

1 .  Maintain  Team  Structure  and  Climate 

2.  Plan  and  Problem  Solve 
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3.  Communicate  with  the  Team 

4.  Manage  Workload 

5.  Improve  Team  Skills 

6.  Integration  Unit 

The  first  module  (“Maintain  Team  Structure  and  Climate”)  included  information  about  the 
composition  of  core  teams,  team  leader  and  team  member  roles,  team  structure,  and  team 
climate.  It  also  presented  a  technique  for  managing  conflict:  the  DESC  Script,  which  encourages 
(describing  the  situation,  expressing  concerns  about  the  action,  .suggesting  alternatives,  stating 
consequences,  and  obtaining  consensus. 

The  “Plan  and  Problem  Solve”  module  taught  skills  such  as  planning,  using  shared  mental 
models,  cross-monitoring,  using  assertion  and  advocacy,  and  using  the  two-challenge  rule. 
Planning  consisted  of  both  long-term  and  situational  planning,  such  as  responding  to 
emergencies.  Shared  mental  models  ensure  that  team  members  have  the  same  understanding 
about  the  situation  and/or  problem.  The  remaining  discussion  focused  on  encouraging  team 
members  to  voice  their  concerns  in  order  to  prevent  errors. 

“Communication”  was  the  theme  for  the  third  module.  This  module  taught  information  about 
situational  awareness  and  the  standards  of  effective  communication  (e.g.,  clear,  timely,  complete, 
and  verified).  Information  transfer  skills  that  encourage  clarification  of  information,  such  as 
check-backs,  call-outs,  and  hand-offs,  are  also  part  of  the  communication  module. 

“Managing  Workload”  was  the  fourth  section.  This  module  focused  on  workload  information 
and  skills  for  managing  workload,  such  as  resource  management,  prioritization,  delegation,  and 
task  assistance. 

The  “Improving  Team  Skills”  module  consisted  of  information  about  performance  goals  and 
feedback,  including  characteristics  of  effective  feedback.  Performance  improvement  skills,  such 
as  teamwork  reviews  conducted  at  the  end  of  a  clinical  event  or  near  the  end  of  a  shift, 
situational  teaching,  and  peer  coaching,  were  explained.  Situational  learning  involves 
encouraging  questions,  while  peer  coaching  consists  of  monitoring  team  members’  performance 
and  instructing  team  members. 

Finally,  the  last  unit  was  the  “Integration  Unit.”  The  purpose  of  this  module  was  to  discuss 
implementation  issues  such  as  staff  expectations  of  implementation,  team  implementation  plans, 
and  sustainment  issues,  such  as  the  strengths  and  weaknesses  of  teamwork  systems  and  action 
plans  for  managing  obstacles  to  implementation.  In  addition,  the  curriculum  contained  a 
teamwork  simulation,  in  which  the  class  was  divided  into  a  simulation  team  and  an  observation 
team.  The  instructor  selected  the  scenario  from  a  Simulation  Matrix,  the  simulation  team  enacted 
the  scenario  using  teamwork  skills,  and  the  observation  team  then  rated  the  team  on  its  teamwork 
actions.  Time  permitting,  students  then  changed  teams  and  enacted  another  scenario.  This 
module  was  designed  to  be  an  open  discussion  among  the  students;  however,  the  simulation  was 
cut  short  or  not  done  in  many  of  the  classes  due  to  time  constraints. 
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Instructor  Selection,  Training,  and  Preparation 

Description.  As  mentioned  previously,  MedTeams™  uses  a  train-the-trainer  approach  to 
implement  the  training.  Individuals  designated  by  their  facility  receive  comprehensive  training. 
DRC  requests  that  designated  instructors  be: 

•  Viewed  as  advocates  of  teamwork 

•  Members  of  a  physician/nurse/technician  training  team 

•  Viewed  as  leaders  among  their  peers  and  service  administrators 

•  In  positions  that  allow  flexibility  in  scheduling 

At  the  end  of  the  train-the  trainer  course,  these  individuals  are  certified  as  MedTeams™ 
instructors.  Certification  requires  instructors  to  complete  MedTeams  training,  complete  the 
Instructor  Certification  Course,  complete  the  practice  teaching  and  coaching  prerequisites,  and 
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pass  a  written  exam  with  a  score  of  no  less  than  80  percent.  MedTeams  instructors  are  then 
responsible  for  implementing  MedTeams  training  at  their  health  care  organization. 

Findings.  Observations  of  the  train-the-trainer  course  in  Boston  and  a  review  of  the  L&D 
instructor  manual  indicated  that  MedTeams  uses  a  variety  of  control  mechanisms  to  ensure 
instructor  quality.  These  include  standards  for  instructor  selection,  an  instructor  certification 
course,  opportunities  to  practice  and  receive  feedback  on  teaching  and  coaching,  and  an 
assessment  of  team  and  course  knowledge  through  a  written  test.  We  conclude  from  reviewing 
this  information  that  this  train-the-trainer  strategy  should  produce  knowledgeable  instructors  who 
provide  high  quality,  reliable  training  at  their  facilities. 
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To  cross-validate  our  finding,  we  interviewed  14  MedTeams  -certified  instructors  who 
conducted  training  at  the  naval  medical  center  we  visited.  As  part  of  these  interviews,  we  asked 
these  individuals  how  they  were  selected  and  how  they  were  prepared  to  teach  the  OR  course. 
Findings  from  these  interviews  are  presented  in  the  following  paragraphs. 

First,  regarding  instructor  selection,  interviewees  provided  a  variety  of  reasons  as  to  why 
they  were  selected  to  teach  the  MedTeams™  curriculum  for  the  OR.  For  example,  interviewees 
reported  that  department  heads  were  required  to  teach  at  least  one  session  of  the  course,  but  a 
few  department  heads  delegated  their  responsibility  to  other  people.  Some  instructors  thought 
that  senior  physicians  were  chosen  to  teach  the  course  in  order  to  encourage  attendees  to  focus 
on  the  course.  A  few  thought  that  they  were  chosen  at  random;  others  thought  that  they  were 
selected  due  to  previous  training  experiences.  Two  individuals  felt  that  they  were  chosen  because 
they  are  the  continuous  improvement  (Cl)  representatives  for  their  de^rtments.  Two  instructors 
were  involved  with  the  development  of  the  OR  version  of  MedTeams™;  thus,  they  were  willing 
to  participate. 

Regarding  preparation,  instructors  spent  a  variety  of  time  periods  preparing  to  teach  the  OR 
course.  Most  spent  between  2  and  6  hours,  with  several  spending  8  hours  or  more  in  planning. 
Most  instructors  spent  between  2  and  4  hours  as  a  group  preparing  to  conduct  the  class,  and  then 
spent  additional  time  reviewing  the  curriculum  on  their  own. 

Besides  the  MedTeams™  instructor  certification  course,  there  was  no  additional  training  or 
preparation  required  to  teach  the  course.  However,  even  with  the  extensive  activities  included  in 
the  certification  course,  most  instructors  did  not  feel  adequately  prepared  to  conduct 
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MedTeams  training.  Several  mentioned  that  they  were  not  aware  that  they  would  become 
instructors  for  the  course  until  the  end  of  the  train-the-trainer  session,  even  though  at  the  onset  of 
the  train-the-trainer  course  they  were  notified  that  they  would  ultimately  be  teaching 
MedTeams  .  Most  felt  that  they  needed  more  time  to  become  familiar  with  the  materials; 
several  mentioned  that  they  did  not  receive  the  materials  until  the  end  of  the  week  before  they 
were  supposed  to  teach  the  course.  In  contrast,  another  group  of  instructors  felt  well-prepared, 
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but  they  also  had  familiarity  with  MedTeams  prior  to  teaching  the  course. 

Instructional  Strategies 

__  *T*\yf 

Description.  MedTeams  purports  to  employ  a  variety  of  training  methods  that  address  two 
of  the  three  recommended  phases  of  CRM  training:  Awareness  and  Practice-and-Feedback.  The 
Continual  Reinforcement  phase  is  not  directly  addressed,  although  it  could  be  argued  that  this 
occurs  through  the  on-going  practice  of  MedTeams  training  and  the  implementation  of  several 
sustainment  strategies. 

Regarding  awareness,  this  phase  includes  8  hours  of  classroom  instruction,  1  hour  of 
behavioral  modeling  using  videotaped  vignettes,  and  1  hour  of  integration/synthesis.  Once  the 
classroom  training  portion  of  the  course  is  complete,  each  team  member  participates  in  a  4-hour 
practicum  that  involves  practicing  teamwork  behaviors  and  receiving  feedback  from  a  trained 
instructor  (Practice  and  Feedback  Phase).  Coaching,  mentoring,  and  review  sessions  are  also 
provided  during  regular  work  shifts.  The  post-classroom  component  of  training  lasts  for 
approximately  6  months. 

Findings.  We  observed  portions  of  1 1  classes  during  our  visit  to  the  naval  medical  center 
during  summer  2003.  Classes  varied  from  3  to  7  hours.  Classes  varied  primarily  as  a  function  of 
the  instructor  and  how  that  individual  chose  to  conduct  the  course.  For  example,  some  instructors 
simply  read  the  slides  and  did  not  add  much  detail  or  explanation.  Other  instructors  used  a  more 
interactive  style,  engaging  the  attendees  in  discussions,  obtaining  examples  from  their 
specialties,  and  even  calling  on  individuals  when  necessary. 

Most  instructors  utilized  the  vignettes  to  demonstrate  key  points  during  training;  a  few 
instructors  even  customized  or  wrote  new  vignettes  tailored  to  their  specialty  (e.g. 
ophthalmology).  Most  instructors  showed  the  videos.  However,  few,  if  any,  instructors  actually 
conducted  the  practice  exercises.  The  primary  reason  for  skipping  these  activities  was  the  lack  of 
time. 

In  addition  to  our  observations,  we  asked  students  to  indicate  the  extent  to  which  they  agreed 
with  the  statement,  “The  training  was  well  organized.”  Somewhat  in  contrast  to  what  we 
observed,  96  percent  of  the  students  agreed  or  strongly  agreed  with  this  statement  (see  Table  4). 
The  vast  majority  of  students  also  reported  that  the  training  content  was  appropriate  for  their 
department. 

In  summary,  it  should  be  noted  that  we  were  only  able  to  view  the  in-class  portion  of 
MedTeams  training.  Observations  and  other  data  were  not  collected  on  any  posttraining 
implementation  strategies.  Regarding  the  classroom  component,  our  observations  and  instructor 
interviews  suggest  that  there  was  wide  variation  in  how  instructors  implemented  MedTeams™ 
training.  Overall,  classes  varied  from  3  to  7  hours.  Even  though  instructors  were  trained  and 
certified,  how  the  course  was  implemented  and  conducted  was  left  to  the  discretion  of  the 
instructors  and  the  quality  of  instruction  varied  considerably.  The  instructional  strategies  that  we 
observed  (lectures,  case  studies,  and  video  demonstrations),  when  they  were  implemented 
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properly,  seemed  to  be  effective  at  achieving  the  desired  objectives.  However,  because  one  of  the 
core  objectives  of  MedTeams™  is  to  enhance  the  team  skills  of  the  participants,  we  would  have 
liked  to  see  more  instructors  implement  the  role-play  exercises.  Nonetheless,  student  reactions  to 
the  organization  of  the  course  and  its  appropriateness  were  extremely  positive. 

Training  Effectiveness 

Description.  The  classroom-based  phase  of  MedTeams™  primarily  relies  upon  the 
collection  of  trainee  reactions  (Kirkpatrick’s  Level  I  data)  to  determine  training  effectiveness. 

The  Emergency  Team  Coordination  Course®  Evaluation  Form  (©  1997  Dynamics  Research 
Corporation)  is  used  for  this  purpose.  For  the  OR  course,  trainees  were  asked  to  rate  the  extent  to 
which  MedTeams  training  achieved  its  purpose  and  goals,  met  its  stated  objectives,  and  was 
well  organized.  In  addition,  participants  were  asked  to  assess  the  quality  of  the  instruction.  One 
item  addressed  the  expertise  of  the  instructor,  and  the  other  addressed  the  appropriateness  of  the 
instructional  strategies  for  achieving  the  desired  objectives. 
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Findings.  To  assess  MedTeams  effectiveness,  we  collected  additional,  independent  data 
on  both  trainee  and  instructor  reactions  to  the  course.  Utility  reaction  data  were  collected  from 
participants  and,  as  mentioned,  interviews  were  conducted  with  instructors.  In  addition,  we 
reviewed  the  existing  empirical  literature  that  has  been  reported  regarding  Kirkpatrick’s  Level  II 
(Knowledge),  Level  III  (Behavior),  and  Level  IV  (Results).  Specifically,  we  reviewed  a  quasi- 
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experiment  that  examined  the  effectiveness  of  MedTeams  in  the  ED  that  was  conducted  by 

Morey  and  colleagues.  Below,  we  describe  our  findings  from  these  activities. 

Level  I  data.  Participants  in  the  OR  course  responded  favorably  to  the  training.  Overall, 
participants  liked  the  content  of  the  course  and  felt  that  it  was  useful.  The  course  met  their 
expectations,  in  part,  due  to  the  content  and  issues,  with  respect  to  departmental  problems  and 
communication  problems  that  were  raised.  More  specifically,  attendees  felt  the  content  was  well- 
organized  (96  percent  agreement;  see  Table  5)  and  appropriate  for  their  department  (92  percent 
agreement).  Almost  all  attendees  felt  confident  that  they  understood  the  material,  could  perform 
teamwork  tasks,  and  could  apply  the  course  material  to  their  jobs.  Though  these  ratings  also  were 
high,  there  was  slightly  less  agreement  that  the  training  was  an  effective  use  of  time  (77  percent) 
and  that  the  training  prepared  attendees  to  work  effectively  in  their  jobs  (78  percent). 
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Table  5.  Posttraining  opinions  about  MedTeams^^ 


Total  Responses  (n-21d) 

Item 

Mean 

Std. 

Dev. 

Percentage 

Agreement 

Percent¬ 

age 

Neutral 

Percentage 

Disagree¬ 

ment 

The  training  was  well-organized. 

4.4 

.68 

96% 

2% 

2% 

1  am  confident  that  1  can  perform 
the  tasks  that  were  trained. 

4.4 

.58 

96% 

3% 

1% 

1  am  confident  that  1  understood 
the  training  content. 

4.5 

.60 

96% 

3% 

1% 

1  am  confident  that  1  can  use  the 
knowledge  that  1  learned  on  the 
job. 

4.4 

.69 

94% 

5% 

1% 

The  training  content  was 
appropriate  for  my  department. 

4.3 

.72 

92% 

6% 

2% 

Training  will  help  my  department 
improve  patient  safety. 

4.1 

.81 

83% 

14% 

3% 

As  a  result  of  this  training,  1  feel 
more  confident  about  my  ability  to 
work  effectively  In  a  team. 

4.1 

.83 

80% 

17% 

3% 

Training  prepared  me  to  work 
effectively  in  my  job. 

4.0 

.82 

78% 

18% 

4% 

Training  was  an  effective  use  of 
my  time. 

3.9 

.95 

77% 

14% 

9% 

When  asked  about  any  changes  that  could  be  made  to  the  course,  a  little  less  than  half  of 
attendees  felt  that  nothing  should  be  changed.  Those  who  felt  that  changes  should  be  made 
wanted  more  interaction  with  others,  including  group  activities  and  role-plays,  and  more  videos, 
vignettes,  and  case  scenarios.  Several  participants  suggested  shortening  the  course,  and  a  few  felt 
that  the  course  was  redundant  in  parts  and  could  be  condensed. 

Almost  all  attendees  would  recommend  this  training  course  to  other  people  at  work.  The 
main  reason  for  recommending  the  course  is  the  need  for  other  co-workers  to  learn  about 
teamwork  and  improve  their  teamwork  skills,  particularly  communication. 

Likewise,  instructors  were  pleased  with  the  course.  They  felt  that  the  sequence  and  flow  of 
the  course  were  appropriate.  The  training  aids  (e.g.,  videos,  vignettes)  were  also  identified  as 
being  useful.  A  few  instructors  noted  that  Module  1  (“Maintain  Team  Structure  and  Climate”) 
appeared  to  be  the  most  important  and  that  the  videos  in  the  beginning  helped  to  gain  trainees’ 
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attention.  However,  a  few  instructors  felt  that  there  were  almost  too  many  videos  and  that  it 
might  be  better  to  include  fewer.  Suggestions  for  other  training  aids  included  more  interactive 
handouts  that  encourage  participation  and  vignettes  with  OR-relevant  examples.  Better  audio¬ 
visual  coordination  was  another  suggestion. 

Instructors  felt  that  the  communication  module  and  the  fmstration  exercise  were  two  of  the 
best  features  of  the  course.  The  frustration  exercise  (i.e.,  having  the  group  voice  their  fhistrations 
with  the  OR)  proved  to  be  very  useful.  Participant  fimstrations  were  written  down  on  a  flip  chart, 
and  several  instructors  linked  the  fhistrations  back  to  specific  components  of  MedTeams™ 
towards  the  end  of  the  course.  This  activity  added  validity  to  the  MedTeams™  training  by 
showing  how  it  can  help  alleviate  or  reduce  some  of  the  attendees’  frustrations.  Other  key 
features  that  were  cited  by  instructors  included: 

•  Pre-packaged  materials  (e.g.,  slides  and  student’s  manual) 

•  Videos 

•  Basic  nature  of  training  (i.e.,  easy  to  understand) 

•  Training  crosses  all  skill  levels/allows  for  face  time  with  all  groups  involved  in  the 
OR/gets  surgeons  involved 

•  Patient-safety  focus  of  training 

•  Identification  of  solutions  to  common  problems 

Instructors  felt  that  less  time  could  be  spent  on  self-explanatory  information.  According  to 
instructors,  specific  components  of  the  course  that  could  use  improvement  include  Modules  4 
and  6  (“Manage  Workload”  and  “Integration”).  Module  4  is  dense  and  difficult  for  many 
participants  to  understand;  several  felt  that  Module  4  (“Workload”)  should  be  revised  and 
shortened.  In  particular,  the  concept  of  “task  assistance”  was  particularly  difficult  for  some 
trainees  to  grasp.  Module  6  does  not  do  a  good  job  of  wrapping  up  the  course,  according  to 
several  instructors.  The  simulations  (i.e.,  role-plays  of  teamwork  skills)  included  in  Module  6 
encourage  lengthy  discussions  instead  of  reviewing  the  key  points  of  the  course. 

Almost  all  instructors  noted  that  the  examples  in  the  course  should  be  made  more  relevant  to 
an  OR.  A  few  instructors  recommended  improving  the  audio-visual  integration;  one  suggested 
including  markers  in  the  text  to  cue  instructors  for  upcoming  videos  and  vignettes.  Other 
recommendations  from  the  instructors  included  the  following: 

•  Include  the  fhistrations  exercise  in  the  curriculum 

•  Re-arrange  the  curriculum  to  put  the  most  important  information  up  front 

•  Consider  the  OR  environment  (some  terms  are  too  “touchy-feely”  or  “hokey”)  and 
address  people’s  roles 

•  Improve  organization  and  planning  for  the  course  (e.g.,  the  lack  of  preparation  time) 

•  Place  greater  emphasis  on  the  following  items: 

°  Integration  (i.e..  How  will  the  training  be  implemented?  How  will  it  work?) 

°  Communication 

°  Improvement  of  team  skills 
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°  OR-based  videos  and  practical  exercises 

°  Involvement  of  the  surgeons  (e.g.,  focusing  on  the  patient  and  working  with  the 

team) 

Finally,  interviewers  asked  the  instructors  what  the  likelihood  is  that  the  training  will  be 
successful  in  their  departments.  Several  believed  that  there  would  be  a  40-50  percent  chance  of 
success  due  to  anticipated  difficulties  with  implementation.  One  of  the  major  concerns  voiced 
was  that  individuals  in  the  OR  do  not  have  enough  time  or  staff  to  keep  the  training  going  and 
ensure  its  successful  implementation.  To  address  this  issue,  one  instructor  suggested  moving  the 
responsibility  for  the  training  course  to  the  Staff,  Education,  and  Training  Group  (SEAT). 
Another  concern  was  that  many  of  the  surgeons  had  not  bought  into  the  training  and  felt  that  it 
was  a  “waste  of  their  time;”  thus,  they  might  not  be  willing  to  implement  it.  A  third  concern  was 
the  length  of  the  training  and  the  need  for  refresher  training.  Additionally,  several  instructors  felt 
that  the  DoD  should  put  more  resources  into  the  training  (e.g.,  organization  and  preparation).  In 
contrast,  a  few  instructors  felt  that  there  was  high  likelihood  that  the  training  would  work,  but  it 
would  depend  on  training  and  sustainment.  According  to  many  instructors,  as  long  as  leadership 
supports  the  training,  it  will  be  implemented,  and  everyone  new  who  comes  in  will  get  the 
training. 

Level  II,  III,  and  IV data.  There  has  been  one  major  evaluation  of  the*MedTeams  approach 

in  emergency  departments  and  another  is  underway  in  labor  and  delivery  units.  The  ED 
study^^  involved  a  multisite,  single-crossover,  quasi-experimental  design.  In  this  study,  nine  EDs 
(six  in  the  experimental  group  and  three  in  the  control  group)  were  observed  during  a  14-month 
interval  that  encompassed  pretraining  baseline  measures,  the  training  intervention  proper,  and 
posttraining  evaluations.  A  suite  of  17  process  and  performance  measures  was  collected.  To 
ensure  rating  accuracy,  all  observation-based  measures  were  collected  by  trained  raters,  and 
measures  of  interrater  agreement  were  periodically  conducted  to  ensure  that  the  raters  remained 
calibrated.  Finally,  because  data  were  clustered.  Generalized  Estimating  Equations  (GEE)  were 
used  to  test  the  effect  of  the  hospital-level  intervention  using  case-level  data.  The  results 
suggested  that,  in  contrast  to  the  control  group,  the  trained  groups  showed  significant  gains  in 
teamwork-related  knowledge,  skills,  and  attitudes;  that  the  intervention  did  not  increase  self- 
reported  task  workload;  and  that  the  error  rate  decreased  sharply. 

The  second  study  is  ongoing  in  L&D  units  in  civilian  and  military  hospitals.^^  Unlike  the 
previous  study,  in  which  the  EDs  chose  to  participate  in  either  the  experimental  or  control 
conditions,  this  study  was  designed  as  a  randomized  clinical  trial.  Based  on  an  a  priori  power 
analysis,  24  hospital  L&D  units  were  randomly  assigned  to  participate  in  either  the  experimental 
or  control  conditions  (up  to  12  per  condition).  Many  other  aspects  of  the  L&D  study  mirror  that 
of  Simon  and  colleagues^"^  with  multiple  performance  measures  that  focus  on  patient  outcomes, 
team  process,  and  staff  and  patient  satisfaction.  However,  unlike  the  ED  investigation, 
performance  ratings  on  the  team  skills  taught  in  MedTeams  will  not  be  collected.  Rather  proxy 
measures,  such  as  the  time  it  takes  for  new  patients  to  be  processed  through  hospital  admissions 
and  the  time  interval  between  deciding  to  do  a  C-section  and  initial  incision,  will  be  recorded. 
Data  collection  for  the  L&D  study  was  expected  to  be  completed  in  the  spring  of  2004,  however, 
the  findings  have  not  yet  been  published. 

Strengths  and  Weaknesses 

Strengths.  Our  review  of  the  literature,  observations  of  the  classroom  phase,  instructor 
interviews,  and  posttraining  assessment  suggest  that  MedTeams™  has  a  number  of  desirable 
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qualities.  First,  the  original  courseware  for  the  ED  was  based  on  a  thorough  up-ffont  needs 
analysis.  This  analysis  resulted  in  the  five  core  dimensions  on  which  MedTeams™  training  is 
based.  Second,  MedTeams  employs  a  very  practical  system  for  implementing  the  training. 
Designated  staff  members  from  a  specific  facility  are  trained  and  certified  as  MedTeams™ 
instructors,  and  then  these  staff  members  conduct  MedTeams™  training  at  their  facilities  (i.e.,  a 
train-the-trainer  strategy).  Moreover,  our  review  of  the  “L&D  Team  Coordination  Course”  for 
instructors  suggests  that  instructor  training  is  comprehensive  and  thorough*.  Third,  our 
independent  collection  of  posttraining  reactions  suggests  that  participants  had  positive  reactions 
to  MedTeams™  training.  Participants  indicated  that  the  training  was  well  organized,  and  they 
felt  that  they  could  use  many  of  the  strategies  that  were  discussed  during  training  upon  returning 
to  their  jobs.  Finally,  the  MedTeams  developers  have  made  the  most  extensive  efforts  to 
collect  Level  II  and  III  data  to  demonstrate  the  effectiveness  of  MedTeams™. Although  the 
initial  investigation  suffered  several  design  flaws,  which  make  the  results  somewhat  tentative,  it 
was  one  of  the  few  efforts  in  the  academic  literature  to  link  team  process  to  health  outcomes. 
Hopefully,  the  L&D  study  will  shed  additional  light  on  these  relationships  and  the  utility  of 
MedTeams™. 

_  TTVyf  _ 

Weaknesses.  Nevertheless,  MedTeams  does  have  its  limitations.  First,  although  a 
comprehensive  needs  analysis  was  performed  to  develop  the  ED  curriculum,  no  subsequent  in- 
depth  needs  analyses  were  conducted  to  develop  the  L&D  and  OR  courses.  Essentially,  subject 
matter  experts  reviewed  the  training  materials  and  customized  the  case  studies  and  other 
examples  when  appropriate.  We  question  this  approach,  because  a  panel  of  leading  experts  in  the 
field  that  was  convened  by  AIR  in  January  2002  suggested  that  team  knowledge,  skills,  and 
attitudes  are  likely  to  vary  by  medical  specialty  as  well  as  other  factors  (see  Appendix  D). 
Second,  we  were  surprised  by  the  variation  in  how  the  classroom  phase  was  administered  by 
trained,  certified  MedTeams  instructors.  Classes  ranged  from  3  to  7  hours  and  the  quality  of 

TTVyf 

this  instruction  varied  greatly.  Despite  the  MedTeams  developers’  best  efforts  to  ensure 
consistency  of  instruction,  consistency  was  lacking.  Also,  while  one  of  the  main  objectives  of 
MedTeams™  is  to  develop  team  skills,  much  of  the  classroom  instruction  focused  on  mastering 
declarative  and  some  procedural  knowledge.  There  was  substantially  less  time  devoted  to  skills 
practice.  Third,  MedTeams  did  not  employ  a  cultural  assessment/evaluation  component  prior 
to  implementing  the  training.  As  a  result,  it  is  entirely  possible  that  MedTeams  is  effective 
only  in  hospitals  that  have  already  made  a  commitment  to  teamwork,  secured  upper-level 
management  support,  established  an  open,  nonpunitive  atmosphere  that  embraces  errors  as  an 
opportunity  for  learning,  and  recognized  the  need  for  change.  It  is  interesting  to  note  that  results 
from  our  pretraining  assessment  suggest  that  this  particular  naval  medical  center  has  a  culture 
that  supports  teamwork.  Fourth,  trainee  reactions  to  MedTeams™  were  positive  despite  the 
tremendous  variability  in  instruction.  However,  and  quite  in  contrast  to  the  positive  reactions, 
several  of  the  instructors  we  interviewed  said  that  there  was  only  a  40-50  percent  chance  of  this 
training  being  successful  when  implemented.  Finally,  one  of  the  limitations  of  MedTeams  is 
the  delay  in  the  implementation  of  the  actual  strategies.  All  departmental  staff  must  receive  the 
classroom  phase  before  implementation.  Although  this  appears  to  be  a  reasonable  approach, 
especially  when  training  a  large  number  of  people,  there  can  be  considerable  delay  between 
classroom  training  and  implementation.  For  example,  we  visited  the  naval  medical  center  again 
in  early  July  2003.  At  that  time,  OR  staff  training  had  not  been  completed.  Such  a  delay  could 
result  in  a  decay  of  important  knowledge  and  strategies  that  were  developed  during  training. 
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Summary 

Trx  n 

In  conclusion,  the  MedTeams  course  was  well  received.  The  content  covered  the  basics  of 
teamwork  and,  for  the  most  part,  students  and  instructors  felt  that  it  delivered  a  good  message: 
teamwork  is  important.  The  course  met  students’  expectations,  and  most  would  recommend  the 
class  to  coworkers.  The  frustration  exercise  was  identified  as  the  most  popular  activity  in  the 
class.  Both  students  and  instructors  liked  the  interaction  between  physicians,  nurses,  and  medical 
technicians  that  resulted  from  the  exercise.  They  felt  that  it  was  a  good  way  to  raise  awareness 
about  issues  and  concerns  from  different  health  care  professionals’  perspectives.  It  also  tied  in 
nicely  with  the  modules  of  the  course,  as  many  of  the  concerns  were  addressed  in  the  modules. 
However,  instructor  variability  plagued  the  actual  conduct  of  the  training.  Moreover,  many 
attendees  were  frustrated  by  the  length  of  the  course  as  well  as  the  over  reliance  on  lectures  as 
the  primary  instructional  strategy.  Many  would  prefer  to  break  the  lecture  portion  up  with  more 
videos,  case  scenarios,  and  group  discussions.  In  addition,  many  wanted  more  examples  that 
were  directly  relevant  to  the  OR. 
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Chapter  3.  Case  Study  2 — Medical  Team  Management* 

Introduction 

The  primary  purpose  of  Medical  Team  Management  (MTM)  is  to  reduce  medical  errors 
through  interdisciplinary  team  training.  MTM  was  modeled  on  the  U.S.  Air  Force’s  CRM 
training  program  for  fighter  pilots  and  was  developed  after  poor  teamwork  was  identified  as  the 
root  cause  of  a  medical  event  that  led  to  a  catastrophic  patient  outcome. 

The  MTM  training  program  has  two  major  components:  a  3 -day  train-the-trainer  course  and 
a  medical  treatment  facility  course.  Upon  completing  the  train-the-trainer  course,  graduates 
return  to  their  respective  medical  facilities  to  train  the  remaining  staff  in  teamwork  principles.^^ 
The  MTM  curriculum  includes  an  introduction  to  the  program,  overviews  of  key  patient  safety 
and  CRM  issues,  and  specific  modules  for  seven  foundational  elements:  leadership,  workload 
performance,  policy  and  regulations,  situational  awareness,  available  resources,  communication 
and  operating  strategy.  In  addition  the  curriculum  includes  obstacles  to  effective  teamwork,  and 
tools  (behaviors)  for  improved  teamwork  and  communication.  Case  studies,  vignettes,  and  tools 
(e.g.,  the  “two  attempt”  rule)  are  interspersed  throughout  the  curriculum  to  reinforce  the 
importance  of  effective  teamwork. 

Case  Study  Approach 

We  were  able  to  access  several  sources  of  information  to  assess  MTM.  Specifically,  we 
reviewed  the  course  materials  for  the  MTM  train-the-trainer  course,  the  MTM  Handbook,  and 
the  MTM  implementation  guidelines.  In  addition  to  our  document  review,  we  observed  the  3 -day 
train-the-trainer  course,  collected  pretraining  data  on  student  experiences  and  expectations, 
collected  posttraining  data  on  student  reactions  to  MTM,  and  interviewed  several  current  and 
former  participants.  Each  of  these  activities  is  described  in  more  detail  below. 

Document  and  Literature  Review 

As  part  of  a  larger  state-of-the-art  literature  review,  AIR  conducted  a  document  review  on 
medical  team  training  and  specific  medical  team  training  programs,  including  MTM.  Unlike 
MedTeams™,  there  were  no  published  articles,  conference  proceedings,  or  technical  reports  that 
document  the  effectiveness  of  MTM.  However,  we  carefully  reviewed  the  MTM  train-the-trainer 
toolkit,  the  MTM  Handbook,  and  the  MTM  implementation  guidelines,  which  were  provided  to 
us  prior  to  attending  the  train-the-trainer  course. 

Course  Observations 

AIR  staff  members  observed  the  MTM  train-the-trainer  course,  which  took  place  at  a  U.S. 

Air  Force  base  during  summer  2003.  As  described  in  the  MTM  curriculum,  the  train-the-trainer 
course  lasted  3  full  days.  The  first  day  consisted  of  an  introduction  to  MTM,  along  with  modules 
on  the  foundational  elements.  The  second  day  focused  on  the  obstacles  to  effective  teamwork. 


*  Note:  Appendixes  cited  in  this  report  are  provided  electronically  at 
http://www.ahra.aov/Qual/teamtrain/index. html#app . 
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MTM  tools,  case  studies,  videotaped  vignettes,  and  implementation  guidance.  The  final  day 
consisted  of  student-led  case  studies,  a  posttraining  wrap-up,  and  student-led  feedback. 

Pre-  and  Posttraining  Assessment  Tools 

In  addition  to  observing  the  MTM  train-the-trainer  course,  AIR  staff  administered  the  pre- 
and  posttraining  assessment  tools  to  MTM  participants.  The  pretraining  tool  was  designed  to 
assess  experiences  prior  to  training  and  to  gauge  attitudes  and  beliefs  about  teamwork.  The 
posttraining  tool  measured  participant  utility  reactions  to  MTM.^  The  attendees  represented  a 
variety  of  medical  specialties  including  anesthesia,  cardio-thoracic  medicine,  dentistry, 

OB/GYN,  internal  medicine,  physical  therapy,  emergency  medicine,  and  pharmacy. 
Approximately  one-third  of  them  were  currently  involved  in  some  form  of  patient  safety  or 
quality  assurance  programs  at  their  respective  medical  facilities.  Of  the  26  participants  surveyed, 
3  were  physicians,  12  were  nurses,  and  1 1  represented  other  groups  such  as  medical  technicians, 
physical  therapists,  pharmacists,  and  dental  technicians. 

Participant  Interviews 

During  our  visit,  we  also  interviewed  10  individuals,  two  of  whom  had  attended  a  previous 
train-the-trainer  course.  Participants  were  asked  about  how  they  were  selected  to  teach  MTM, 
their  training  background,  and  their  perceptions  of  the  flow  of  the  course  and  training  aids  which, 
were  used.  In  addition,  they  were  asked  to  name  the  three  best  features  of  the  course  and  three 
that  could  use  improvement,  as  well  as  any  major  obstacles  to  implementation.  Finally,  they 
were  asked  how  they  felt  their  coworkers  might  respond  to  the  MTM  course,  and  when  they 
expected  the  training  to  be  implemented  at  their  respective  medical  facilities  (see  Appendix  C). 

Below  we  present  the  results  of  this  case  study.  These  results  are  organized  around  the  same 
set  of  variables  that  we  used  to  draw  conclusions  about  MedTeams  .  For  each  variable,  we 
provide  descriptive  data  from  our  literature  review  and  then  findings  from  our  observations,  data 
collections,  and  interviews  when  appropriate.  Table  4  (p.  18- 19)  provides  a  summary  of  this 
information  for  each  of  the  three  training  programs  that  we  reviewed. 

Results 

Theoretical  Basis 

The  core  objectives,  curriculum,  and  instructional  strategies  for  MTM  were  derived  from  the 
U.S.  Air  Force’s  aviation  CRM  training  course.  Like  MedTeams  ,  the  MTM  course  developers 
argued  that  medicine  and  aviation  share  a  number  of  similarities,  thereby  making  CRM  training 
for  pilots  portable  to  health  care.  However,  unlike  MedTeams  ,  which  has  been  tailored  to 
specific  medical  specialties  (e.g.,  ED,  OR,  L&D),  MTM  was  designed  to  be  a  generic  course  that 
can  apply  equally  well  in  a  variety  of  medical  specialty  and  support  areas  with  little  or  no 
modification. 

Analysis  of  Training  Needs 

MTM  was  developed  after  an  adverse  event  resulted  in  a  newborn  developing  severe 
neurological  problems.^^  Although  not  part  of  a  formal  needs  assessment,  the  need  for 


^  We  administered  posttraining  tools  after  the  second  day  of  training.  Even  though  this  was  a  train-the-trainer  class 
and  participants  were  future  MTM  instructors,  the  first  2  days  of  the  course  were  devoted  to  demonstrating  MTM 
training. 
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communication  and  team  training  was  validated  by  a  subsequent  review  of  60  closed  cases.  The 
reviewers  identified  poor  communication  as  the  primary  cause  (74  percent)  of  otherwise 
preventable  adverse  events,  providing  further  support  for  MTM.  Drill-down  analyses  suggested 
that  these  60  cases  resulted  in  92  separate  errors.  Furthermore,  in  cases  where  communication 
was  an  issue,  there  was  an  average  of  two  errors  per  case.^ 

Training  Objectives 

The  primary  purpose  of  MTM  is  to  reduce  medical  errors  by  teaching  human  factors 
concepts  to  interdisciplinary  teams  of  medical  professionals. A  secondary  purpose  is  to 
change  the  military’s  medical  culture.  Traditionally,  the  culture  has  focused  on  individual 
performance  and,  in  doing  so,  has  created  obstacles  to  communication.  MTM  was  specifically 
developed  to  foster  a  culture  that  values  team  performance  and  encourages  effective 
communication  across  medical  specialty  areas  and  throughout  the  chain  of  command. 

Training  Content 

The  MTM  training  program  covers  seven  foundational  elements: 

1 .  Leadership 

2.  Workload  Performance 

3.  Policy  and  Regulations 

4.  Available  Resources 

5.  Situational  Awareness 

6.  Communication 

7.  Operating  Strategy 

In  addition,  obstacles  to  effective  communication  and  teamwork  and  tools,  or  desired 
behaviors,  are  critical  elements  taught  in  MTM  .  All  are  designed  to  improve  participants’ 
knowledge  and  skills  in  these  core  areas. 

The  “Leadership”  module  highlights  the  differences  between  leadership  and  authority,  and 
provides  techniques  for  regulating  information  flow,  directing  team  activities,  motivating  team 
members,  and  making  effective  team  decisions.  The  “Workload  Performance”  module  highlights 
the  curvilinear  effect  of  workload  on  performance — that  performance  worsens  when  workload  is 
either  too  high  or  too  low.  It  also  provides  techniques  for  identifying  and  dealing  with  high  levels 
of  workload,  distraction,  stress,  and  fatigue.  The  “Policy  and  Regulations”  module  focuses  on  the 
importance  of  following  guidelines  for  maintaining  patient  safety,  the  various  reasons  for 
departing  from  established  guidelines,  and  the  consequences  of  doing  so.  The  “Available 
Resources”  module  discusses  using  all  assets,  such  as  team  members,  equipment,  skills,  and  all 
information  available  to  the  medical  team.  It  describes  both  internal  and  external  resources  and 
emphasizes  that  thorough  and  effective  use  of  these  resources  maximizes  safety.  The  “Situational 
Awareness”  (SA)  module  defines  the  concept  of  situational  awareness,  provides  cues  to 
determine  if  team  members  have  lost  SA,  and  suggests  techniques  for  maintaining  SA.  The 
“Communication”  module  identifies  the  various  forms  of  verbal  and  nonverbal  communication. 

It  also  identifies  principles  for  effective  communication,  obstacles  to  effective  communication, 
and  real-life  examples  of  communication  problems  between  health  care  providers.  The 
“Operating  Strategy”  module  provides  recommendations  for  integrating  the  various  teamwork 


31 


issues.  It  also  highlights  the  role  of  shared  mental  models,  which  develop  as  team  members 
interact  over  time  and  shows  how  shared  mental  models  can  improve  team  performance.  The 
“Obstacles  to  Effective  Teamwork”  module  identifies  a  number  of  factors — such  as  excessive 
professional  courtesy,  the  halo  effect,  and  hidden  agendas — that  cause  breakdowns  in  team 
communication  and  set  the  stage  for  medical  errors.  The  final  module  teaches  tools,  or  desired 
behaviors  such  as  the  “FM  SAFE  checklist,”  “assertive  statement,”  and  “two  attempt”  rule  for 

combating  obstacles  and  improving  teamwork. 

1 

Instructor  Selection,  Training  and  Preparation 

Description.  Our  review  of  MTM  documents  found  that  the  U.  S.  Air  Force  requests  that 
candidate  MTM  instmctor  have  at  least  5  years  of  clinical  experience  in  their  specialty  areas.  In 
addition,  instructors  should  have  at  least  1  year  of  retainability  in  the  armed  forces,  must  be  a 
competent  speaker,  and  are  expected  to  have  previous  experience  delivering  training. 

Findings.  Although  MTM  has  specific  requirements  for  MTM  instructors,  we  could  not 
identify  any  formal  procedure  for  selecting  potential  MTM  trainers.  Our  interviews  with  the 
MTM  participants  suggested  that  most  were  either  recommended  by  a  senior  officer,  or  had 
volunteered  because  of  their  personal  interest  in  maintaining  patient  safety.  The  participants 
came  from  a  variety  of  medical  specialties  (from  surgeons  to  medical  technicians),  and  included 
enlisted  to  mid-level  officers  (from  E-5s  to  0-6s).  This  mix  was  not  unexpected  because  MTM 
was  designed  to  be  an  interdisciplinary  training  course,  which  encourages  open  and  effective 
communication,  regardless  of  rank  and  content  domain. 

Instructional  Strategies 

Description.  MTM  employs  a  variety  of  training  methods.  These  include  computer-based 
instruction  (which  has  temporarily  been  suspended),  didactic  lectures,  seminar  participation, 

o  ^  ^  /I 

application  questions,  behavioral  modeling,  and  case  studies.  ’  ’  The  trainees  are  also  required 
to  complete  a  variety  of  homework  assignments.  One  involved  observing  one’s  own  team  to 
identify  obstacles  that  hinder  effective  team  performance.  Another  requires  the  trainees  to 
practice  the  tools  that  they  have  learned  in  the  workplace.  The  trainees  then  identify  the  lessons 
that  they  have  learned  and  discuss  them  at  subsequent  training  sessions.^^ 

Findings.  During  the  train-the-trainer  course,  we  observed  didactic  lectures,  seminar 
participation,  application  questions,  and  case  study  analyses.  The  case  studies  were  a  significant 
portion  of  the  class,  as  the  instructors  went  over  the  technique  on  the  second  day  and  assigned 
students  to  develop  their  own  case  study  for  the  next  day.  On  the  third  day,  the  participants 
discussed  their  case  studies  in  small  groups.  The  participants  generally  seemed  to  enjoy  this 
exercise,  and  many  recommended  including  more  case  studies  in  future  MTM  train-the-trainer 
courses  (see  Table  6). 

A  number  of  techniques  for  sustaining  and  reinforcing  the  human  factors  concepts  discussed 
were  also  included  in  MTM  training.  For  example,  a  module  on  the  third  day  was  devoted  to 
sustainment  issues,  especially  in  unanticipated  situations.  Topics  included  long-term  planning, 
briefings,  and  continuous  monitoring  of  operations. 

In  summary  it  should  be  noted  that  we  were  only  able  to  view  the  in-class  portion  of  the  3- 
day  train-the-trainer  course  for  MTM.  Therefore,  unlike  MedTeams™  and  DOM,  trainees  were 
future  MTM  instructors  not  participants.  Our  observations  suggested  that  the  first  day  of  MTM 
training  was  an  orientation  day;  essentially  instructors  received  MTM  training.  Therefore,  we 
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believe  that  the  results  we  collected  on  our  posttraining  measure  are  representative.  With  that  in 
mind,  we  now  turn  to  these  findings. 

Training  Effectiveness 

Description.  The  MTM  instructors  administered  an  evaluation  form  at  the  end  of  the  training 
course.  The  form  included  demographic  information  such  as  participants’  position,  employment 
status  (i.e.,  active  duty  or  civilian),  and  experience  level.  Participants  were  then  asked  to  rate  the 
various  pieces  of  the  course  and  the  trainer  toolkit,  using  a  four-point  scale,  with  anchors  ranging 
from  “not  helpful  at  all”  (1)  to  “very  helpful  ”  (4).  They  also  rated  the  effectiveness  of  the 
instructors  and  gave  an  overall  course  rating,  using  a  three-point  scale,  with  anchors  ranging 
from  “marginally  effective”  (1)  to  “very  effective”  (3).  In  addition,  they  were  asked  to  rate  their 
self-efficacy  at  teaching  and  applying  MTM  principles  in  their  respective  medical  facilities  using 
a  four-point  scale,  with  anchors  ranging  from  “very  uncomfortable”  (1)  to  “very  comfortable” 

(4).  Finally,  they  were  asked  to  identify  the  one  thing  that  was  most  helpful  and  the  one  thing 
that  was  least  helpful  in  preparing  for  the  course. 

Findings.  Level  I  data.  Overall,  the  MTM  course  was  well  received.  Participant  reactions 
indicated  that  it  was  well  organized  and  contained  appropriate  content.  Many  attendees  exuded 
confidence  from  the  training  course;  they  felt  confident  that  they  understood  the  material  and 
could  perform  the  teamwork  tasks  and  apply  them  to  their  jobs  (see  Table  6).  The  course  met 
their  expectations  because  it  clearly  presented  teamwork  information  and  the  goals  of  the 
program.  In  addition,  it  gave  attendees  skills  to  use  to  help  improve  teamwork  in  their 
departments. 

Attendees  were  also  pleased  with  the  sequence  of  the  course.  They  were  particularly  pleased 
that  the  material  was  presented  and  then  followed  by  examples  and/or  videos  to  support  it.  The 
videos,  vignettes,  and  case  studies  helped  reinforce  the  material  for  the  attendees.  However, 
many  of  the  videos  and  vignettes  were  examples  of  poor  teamwork  skills;  attendees  noted  that 
examples  of  excellent  teamwork  skills  should  also  be  included.  In  fact,  one  attendee  suggested 
having  each  facility  participating  in  the  training  submit  a  teamwork  success  story  ahead  of  time 
that  could  be  discussed  at  the  course  in  order  to  emphasize  the  results  of  effective  teamwork. 
However,  it  could  be  difficult  for  attendees  to  pinpoint  such  examples. 

Attendees  felt  that  the  three  major  strengths  or  best  features  of  the  course  include  the 
material,  particularly  the  communication  section,  the  videos,  vignettes,  and  case  studies,  and  the 
speakers.  They  also  liked  the  audience  participation  and  interaction  both  within  the  smaller 
groups  and  with  the  instructors.  Other  features  that  were  named  include  the  templates  in  the 
manual  and  the  online  portion  of  the  course,  where  students  are  able  to  go  through  some  of  the 
material  at  their  own  pace  before  they  attend  the  course.  Unfortunately,  the  online  training  aid  is 
not  currently  available.  Several  attendees  would  like  to  have  the  online  section  available  as  both 
a  preview  of  the  information  for  those  who  will  be  attending  the  class  and  also  as  refresher 
training  for  those  who  have  already  taken  the  course. 
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Table  6.  Posttraining  opinions  about  MTM 


Total  (n=26) 

Item 

Mean 

Std. 

Dev. 

Agreement 

Neutral 

Disagreement 

The  training  was  well  organized. 

4.6 

.49 

1 00% 

- 

- 

1  am  confident  that  1  understood 
the  training  content. 

4.5 

.59 

96% 

4% 

- 

1  am  confident  that  1  can  perform 
the  tasks  that  were  trained. 

4.2 

.72 

92% 

4% 

4% 

1  am  confident  that  1  can  use  the 
knowledge  that  1  learned  on  the 
job. 

4.3 

.80 

88% 

8% 

4% 

As  a  result  of  this  training,  1  feel 
more  confident  about  my  ability  to 
work  effectively  In  a  team. 

4.2 

.65 

88% 

12% 

The  training  content  was 
appropriate  for  my  department. 

4.2 

.75 

88% 

8% 

4% 

Training  will  help  my  department 
Improve  patient  safety. 

4.1 

.60 

88% 

12% 

- 

Training  was  an  effective  use  of 
my  time. 

4.1 

.70 

88% 

8% 

4% 

Training  prepared  me  to  work 
effectively  In  my  job. 

4.0 

.71 

76% 

24% 

- 

Nearly  all  attendees  would  recommend  the  course  to  their  fellow  co-workers,  because  they 
felt  that  everyone  could  benefit  from  teamwork  training.  In  addition,  the  course  emphasizes 
critical  skills  that  are  vital  for  safe  and  effective  patient  care.  However,  two-thirds  of  attendees 
would  change  the  course  if  they  were  to  re-design  it.  The  most  popular  suggestions  include 
adding  more  case  studies,  scenarios,  and  videos  as  examples  of  both  poor  and  effective 
teamwork  and  deleting  the  aviation  CRM  videos.  Several  attendees  felt  the  aviation  videos  were 
difficult  to  understand  and  felt  that  explaining  how  CRM  in  aviation  relates  to  medical  team 
training  was  sufficient.  A  few  attendees  felt  that  the  course  was  too  long  and  there  was  too  much 
time  devoted  to  lecture.  Attendees  also  felt  they  needed  more  answers  to  difficult  questions  and 
resistance  they  may  face  with  leadership  upon  their  return  from  the  course  and  attempts  at 
implementation.  A  final  suggestion  made  was  to  change  the  case  study  template  to  ensure  that  it 
matches  the  order  in  which  the  foundational  elements  are  presented  in  the  manual. 

The  two  biggest  obstacles  attendees  named  were  getting  buy-in  from  leadership  and 
scheduling  the  training.  The  difficulty  in  convincing  leadership  stems  from  the  challenge  of 
showing  them  tangible  values  and  benefits  to  the  training,  such  as  a  decrease  in  the  number  of 
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medical  errors  made  or  a  large  amount  of  money  saved  due  to  the  training.  With  regards  to 
scheduling,  it  is  difficult  to  make  arrangements  for  numerous  people  from  the  same  department 
to  attend  an  all-day  course;  it  limits  patients’  access  to  health  care.  Also,  deployments  and 
assignment  changes  make  it  difficult  to  get  everyone  trained.  A  few  people  thought  that  the 
training  should  be  mandatory  to  have  a  real  effect  on  patient  care. 

Level  II,  III,  and  IV  data.  Unlike  MedTeams  ,  we  could  not  identify  any  additional  data  in 
the  published  literature  on  MTM  describing  its  effectiveness  beyond  trainee  reactions. 

Strengths  and  Weaknesses 

Strengths.  To  summarize,  MTM  offers  a  number  of  advantages.  First,  it  uses  a  series  of 
active  learning  techniques — including  didactic  lectures,  behavioral  modeling,  and  case  studies — 
to  develop  trainees’  teamwork-related  knowledge,  skills,  and  attitudes.  Second,  it  leverages 
known  principles  from  human-factors  research.  For  example,  MTM  training  (a)  explicitly 
distinguishes  between  destructive  and  constructive  conflict  resolution,  (b)  recognizes  that  the 
workload-performance  relationship  is  curvilinear,  and  (c)  distinguishes  between  authority  (which 
is  based  on  rank)  and  leadership  (which  is  based  on  skills).  Third,  MTM  training  is 
interdisciplinary  in  nature,  thereby  teaching  physicians,  nurses,  technicians  and  other  key 
constituencies  to  work  together.  Finally,  it  provides  a  reference  list  that  allows  participants  to 
continue  refining  their  teamwork  skills  after  they  have  completed  the  training. 

Weaknesses.  Nevertheless,  like  the  other  two  programs,  MTM  has  disadvantages.  First,  far 
more  of  the  training  time  is  devoted  to  providing  factual  information  than  to  practicing  actual 
skills  with  instructor  feedback;  the  skills  practice  that  is  provided  primarily  involves  low-fidelity 
techniques  such  as  case  studies.  Second,  although  MTM  provides  trainees  with  a  variety  of 
“tools”  to  reinforce  and  sustain  their  teamwork  skills,  many  of  these  aids  are  not  tools  in  the 
strictest  sense  of  the  word.  More  often  than  not,  the  MTM  materials  consist  of  best  practices  or 
procedures  (e.g.,  briefings,  cross-checks),  but  not  tangible  tools  (e.g.,  checklists,  quick  reference 
cards,  etc.)  that  trainees  can  physically  take  with  them.  Third,  even  though  MTM  is  based  on  the 
“train-the-trainer”  paradigm,  it  does  not  appear  to  include  mechanisms  for  preventing 
performance  degradation  among  trainers.  Finally,  there  does  not  appear  to  be  a  formal  recurrency 
module  for  ensuring  the  maintenance  of  trained  knowledge  and  skills. 

Summary 

In  conclusion,  attendees  enjoyed  the  MTM  course.  It  made  them  more  aware  of  teamwork 
issues  and  taught  them  valuable  skills.  The  most  common  suggestion  for  improvement  that 
attendees  made  was  to  include  more  vignettes  and  case  studies  and  even  cut  down  the  lecture  if 
necessary.  Finally,  participants  also  highlighted  the  importance  of  U.S.  Air  Force  leadership 
support  to  ensure  that  MTM  is  successful. 
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Chapter  4.  Case  Study  3 — Dynamic  Outcomes 
Management®* 

Introduction 

The  primary  purpose  of  DOM — ^renamed  LifeWings'^^  subsequent  to  our  study — is  to 
increase  patient  safety,  reduce  medical  errors,  and  improve  the  quality  of  health  care.^^  DOM 
achieves  this  by  improving  trainees’  skills  in  team-building,  recognizing  adverse  situations, 

•  'X'7 

counteracting  the  effects  of  stress  and  fatigue,  communicating,  and  decision-making.  DOM 
provides  interdisciplinary  team  training  to  surgeons,  nurses,  and  anesthesiologists.  The  program 
draws  heavily  on  Crew  Resource  Management  (CRM)  training  from  aviation  ,  and  was 
developed  by  Crew  Training  International  (CTI),  which  offers  specialized  training  programs  for 
aviation,  construction,  general  business  practice,  and  the  medical  industry. 

Case  Study  Approach 

For  DOM,  we  were  able  to  access  several  sources  of  information.  Specifically,  we  reviewed 
all  instructional  materials  for  the  course  that  was  observed.  Relevant  research  studies  that  have 
been  published  on  the  effectiveness  of  DOM  training  were  also  reviewed.  In  addition  to  our 
document  review,  we  observed  two  administrations  of  the  course  at  a  regional  university  medical 
center  and  collected  pretraining  data  on  student  experiences  and  expectations  and  posttraining 
data  on  student  reactions  to  DOM.  Each  of  these  activities  is  briefly  described  in  more  detail 
below. 

Document  and  Literature  Review 

Our  document  review  identified  one  article  that  has  been  published  on  the  DOM  approach. 
This  paper  was  published  by  Rivers  and  colleagues  and  describes  an  evaluation  of  the 
effectiveness  of  this  training.  Results  of  this  paper  are  described  when  we  review  the  additional 
evidence  that  supports  DOM. 

In  addition  to  reviewing  the  literature,  we  carefully  reviewed  the  Student  Guides  from  the 
regional  university  medical  center  course.  CTI  staff  provided  these  guides  to  us  when  we 
attended  and  observed  the  training. 

Course  Observations 

AIR  staff  observed  the  two  sessions  of  DOM  training  that  were  administered  to  hospital  staff 
at  a  regional  university  medical  center  in  September.  Attendees  were  largely  from  trauma, 
emergency  room,  and  cardio-thoracic  medical  areas.  Over  three- fourths  of  the  attendees  were 
nurses  in  addition  to  a  few  physicians,  scrub  technicians,  paramedics,  and  receptionists.  The 
same  two  CTI  staff  members  taught  the  course  each  day.  Instructors  were  retired  Navy  pilots 
who  had  also  flown  commercial  aircraft.  Observations  were  made  of  the  classroom  portion  of 
this  training,  which  lasted  8  hours  each  day. 


*  Note:  Appendixes  cited  in  this  report  are  provided  electronically  at 
http://www.ahra.QOv/qual/teamtrain/index. html#app . 
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Pre-  and  Posttraining  Assessment  Tools 

In  addition  to  observing  the  classes,  AIR  staff  administered  pre-  and  posttraining  assessments 
to  all  DOM  participants.  The  pretraining  tool  was  designed  to  assess  student  experiences  and 
gauge  participant  attitudes  and  beliefs  about  teamwork.  The  results  from  this  measure  were 
described  in  an  earlier  section  of  this  report  entitled,  “Description  of  Participants.”  The 
posttraining  tool  measured  participant  utility  reactions  to  DOM  training. 

Instructor  Interviews 

Unlike  MedTeams  and  MTM,  DOM  instructors  were  not  interviewed.  This  was  primarily 
a  function  of  the  fact  that  DOM  does  not  use  a  train-the-trainer  technique.  All  DOM  instructors 
are  CTI  employees.  Therefore,  many  of  our  interview  questions  regarding  instructor  selection 
and  training  were  irrelevant.  However,  although  no  formal  interviews  were  conducted,  we  did 
have  informal  discussions  with  representatives  from  CTI  about  various  aspects  of  the  course. 

Below,  we  present  the  results  from  our  analysis  of  this  information.  These  results  are 

'T'\  A 

organized  around  the  same  set  of  variables  that  we  used  to  assess  MedTeams  and  MTM.  For 
each  variable  we  provide  descriptive  data  from  our  literature  review  and  then  findings  from  our 
observations,  and  data  collection,  when  appropriate.  Table  4  (p.  18-19)  provides  a  summary  of 
this  information  for  DOM. 


Results 


Theoretical  Basis 

The  core  objectives,  curriculum,  and  instructional  strategies  for  DOM  were  derived  from 
CRM  training  programs  that  CTI  has  developed  for  the  U.S.  Air  Force,  U.S.  Navy,  as  well  as  a 
number  of  other  commercial  airlines  and  military  clients.  The  CTI  developers  believe  that  there 
are  a  number  of  aviation  safety  practices  that  can  be  applied  directly  to  health  care.  Prior  to 
conducting  the  course,  DOM  staff  spends  several  days  at  the  hospital  site  observing  and 
conducting  informal  interviews  with  staff.  Information  gained  from  these  activities  is  used  to 
customize  the  DOM  training  materials  to  the  particular  institution  in  which  the  training  will  be 
administered. 

Analysis  of  Training  Needs 

We  were  unable  to  identify  any  documentation  that  described  how  DOM  was  developed. 
Other  than  its  basis  in  CRM  and  the  fact  that  DOM  developers  spend  time  at  the  institution  in 
which  the  training  will  be  implemented,  we  could  find  no  evidence  of  any  other  needs  analysis 
activities  (e.g.,  closed  case  reviews,  critical  incident  study,  etc.).  We  suspect  that  much  of  this 
training  was  derived  from  existing  course  materials  that  CTI  uses  to  conduct  CRM  training  for 
the  military  and  commercial  airlines.  A  review  of  these  materials  demonstrates  its  strong  basis  in 
aviation.  However,  we  have  no  evidence  to  confirm  or  disconfirm  this  hypothesis. 

Training  Objectives 

The  primary  objective  of  DOM  is  to  teach  aviation  safety  practices  to  health  care  workers. 
The  DOM  developers  believe  that  there  are  many  parallels  between  these  industries  that  make 
CRM  training  relevant.  These  include  stress,  the  need  for  highly  functioning  teams,  the 
importance  of  accurate  and  precise  communications,  and  the  high  cost  associated  with  systems 
failures.^^ 
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Training  Content 

DOM  consists  of  seven  modules  that  are  taught  using  an  8-hour  lecture,  discussion,  and  small 
group  activity  format.  They  are: 

1 .  Introduction 

2.  Managing  Fatigue 

3.  Creating  a  Team 

4.  Recognizing  Adverse  Situations 

5.  Cross-Check  and  Communication 

6.  Decisionmaking 

7.  Performance  Feedback 

The  “Introduction”  module  is  a  relatively  short  module  that  introduces  the  participants  to 
DOM  training,  CRM,  and  the  idea  of  aviation  safety  practices  and  their  application  to  health 
care.  The  core  skills  that  are  the  target  of  DOM  training  are  also  introduced.  These  include  team 
management,  recognizing  adverse  events,  communication,  decisionmaking,  and  performance 
feedback.  Module  2  covers  “Managing  Fatigue.”  Here,  participants  learn  to  recognize  how 
fatigue  can  affect  their  performance  and  specific  countermeasures  for  managing  the  negative 
effects  of  fatigue.  Module  3  covers  the  topic  of  “Creating  a  Team.”  Here,  the  benefits  of 
teamwork  are  presented  and  discussed.  In  addition,  participants  engage  in  a  small  group  activity 
in  which  they  compile  lists  of  things  that  team  leaders  and  team  members  should  and  should  not 
do.  Module  4  focuses  on  “Recognizing  Adverse  Situations.”  Here,  the  notion  of  red  flags  is 
described,  and  participants  learn  what  they  should  do  when  they  observe  a  red  flag  (i.e.,  see  it, 
say  it,  fix  it).  Several  case  studies  are  then  reviewed  as  a  group.  Each  case  study  involves 
identifying  red  flags  and  solutions  to  these  potential  problems.  Module  5  focuses  on  “Cross¬ 
checking  and  Communication.”  Here,  the  steps  in  the  communication  process  are  presented  as 
well  as  the  four  parts  of  an  assertive  statement.  Again,  participants  are  given  an  opportunity  to 
apply  the  information  presented  during  a  case  study  exercise  at  the  end  of  the  module.  Module  6 
covers  “Decisionmaking.”  Here,  the  different  types  of  team  decisions  are  described,  as  is  the 
concept  of  building  a  shared  mental  model.  Participants  then  apply  these  concepts  during  an  in- 
class  role-playing  exercise.  Finally,  Module  7  focuses  on  “Performance  Feedback.”  Here, 
specific  strategies  are  provided  regarding  how  to  debrief  an  individual’s  or  team’s  performance 
as  well  as  request  feedback  jfrom  a  teammate. 

Instructor  Selection,  Training,  and  Preparation 

As  mentioned  previously,  unlike  MedTeams  and  MTM,  which  employ  a  train-the-trainer 
format,  CTI  staff  conducts  DOM  training.  The  two  instructors  who  conducted  the  course  were 
both  retired  Navy  fighter  pilots  and  also  had  worked  for  commercial  air  carriers.  We  did  not  ask 
these  individuals  how  they  were  trained  or  how  much  preparation  was  involved,  but  we  observed 
high  reliability  in  how  the  training  was  conducted.  In  our  opinion,  CTI  instructors  were 
extremely  professional,  well  trained,  and  highly  engaging. 
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Instructional  Strategies 

Description.  DOM  purports  to  employ  a  variety  of  training  methods  that  address  two  of  the 
three  recommended  phases  of  CRM  training:  Awareness  and  Practice-and-Feedback.  The 
Continual  Reinforcement  phase  is  not  addressed. 

The  awareness  phase  includes  8  hours  of  classroom  instruction.  Once  the  classroom-training 
portion  of  the  course  is  complete,  CTI  staff  work  with  hospital  personnel  to  develop  checklists 
and  other  tools  that  can  be  used  to  promote  safety  at  the  hospital  (the  Practice-and-Feedback 
phase).  One  of  the  CTI  trainers  reviewed  the  basic  materials  with  us  and  showed  us  examples 
that  had  been  developed  for  other  clients.  However,  we  never  received  any  copies  of  this 
information  for  a  more  in-depth  review. 

Below,  we  describe  our  findings  from  reviewing  the  classroom  component  of  DOM — 
Awareness  Training.  Although  we  would  have  liked  to  observe  the  actual  applications  of  the 
posttraining  tools  and  checklists,  this  was  not  possible  because  of  time  constraints. 

Findings.  We  observed  two  complete  classes.  Both  classes  started  at  8  o’clock  in  the 
morning  and  concluded  by  4  o’clock  in  the  afternoon.  Classes  were  kicked  off  by  the  Associate 
Dean  for  Clinical  Affairs,  who  emphasized  the  university  health  center’s  goal  of  becoming  the 
Nation’s  safest  hospital.  The  same  two  CTI  instructors  then  conducted  the  training.  Instructors 
were  consistent  in  their  presentation  of  the  materials  across  the  two  sessions.  Effective  use  was 
made  of  the  videotapes  demonstrations,  in-class  exercises,  and  cases  studies,  which  were 
designed  to  support  the  lecture  portion  of  the  training. 

Tnv  >f 

Similar  to  MedTeams  and  MTM,  we  asked  students  to  indicate  the  extent  to  which  they 
agreed  with  the  statement,  “The  training  was  well  organized.”  Overwhelmingly,  students  agreed 
or  strongly  agreed  with  this  statement  (see  Table  7).  The  vast  majority  of  students  also  reported 
that  the  training  content  was  appropriate  for  their  department. 

Training  Effectiveness 

Description.  For  the  classroom-based  phase,  DOM  relies  upon  the  collection  of  trainee 
reactions  (Kirkpatrick’s  Level  I  data)  and  a  Human  Factors  Attitude  Survey  to  determine  training 
effectiveness.  Regarding  the  reaction  measure,  participants  were  asked  to  rate  how  useful  each 
DOM  training  module  was.  In  addition,  participants  were  asked  to  assess  the  quality  of  the 
instruction  provided  by  each  instructor.  Regarding  the  Human  Factors  Attitude  Survey, 
participants  were  asked  to  complete  this  survey  twice,  once  prior  to  training  and  once  after 
training.  This  measure  appears  to  be  modeled  on  the  Cockpit  Management  Attitude  questionnaire 
developed  by  Helmreich  and  colleagues,^  ^  though  we  received  no  direct  information  on  the 
measure’s  development,  its  subscales,  or  its  psychometric  qualities.  However,  CTI  did  not 
withhold  this  information  from  us;  we  merely  did  not  request  it  because  we  viewed  a 
psychometric  analysis  of  this  tool  as  beyond  the  scope  of  the  evaluation. 

Findings.  Similar  to  our  investigation  of  MedTeams™  and  MTM,  we  collected  additional, 
independent  data  on  trainee  reactions  to  the  course.  Utility  reaction  data^^  were  collected  from 
participants.  In  addition,  we  reviewed  existing  empirical  literature  that  has  been  reported 
regarding  Kirkpatrick’s  Level  II  (Knowledge),  Level  III  (Behavior),  and  Level  IV  (Results).  We 
reviewed  an  evaluation  of  the  implementation  of  DOM  that  showed  that  the  trainees  developed 
positive  attitudes  toward  the  importance  of  teamwork,  rated  the  training  as  useful,  and 
demonstrated  a  50  percent  reduction  in  surgical  count  errors.  Below,  we  describe  our  findings 
from  these  activities. 
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Level  I  Data,  Attendees  of  the  DOM  courses  had  positive  reactions  to  the  course.  Not  only 
did  they  like  the  content  and  understand  it  clearly  (98  percent  agreement),  they  also  felt  that  the 
training  course  was  useful.  Almost  all  (96  percent)  felt  that  they  could  use  the  knowledge  they 
learned  on  the  job  and  perform  the  tasks  learned  (95  percent).  Because  of  the  training,  they  also 
felt  more  confident  about  their  ability  to  work  effectively  in  a  team  (92  percent)  (see  Table  7). 
Finally,  the  course  met  most  attendees’  expectations  (98  percent),  mainly  because  of  the 
informative  content  and  the  skills  (e.g.,  working  effectively  as  a  team  and  communication)  that 
were  taught. 

When  asked  if  they  would  make  any  changes  to  the  course,  the  majority  of  attendees  would 
not  change  anything  about  the  course.  Other  attendees  would  like  to  include  more  area-specific 
scenarios  and  practical  applications,  such  as  role-plays,  and  more  information  on  assertiveness 
advice  and  effective  communication  skills  at  different  levels  (e.g.,  RN,  MD,  etc.).  A  few 
attendees  would  like  to  see  a  more  diverse  group  of  students.  For  example,  they  would  like  to  see 
technicians,  nurses,  nurse  practitioners,  and  physicians  from  different  areas  mixed  better  at  the 
different  tables.  In  addition,  a  few  attendees  would  like  to  have  a  re-evaluation  done  in  the  future 
to  learn  if  the  principles  taught  have  been  integrated  and  to  see  changes  in  best  practices  at  other 
institution  following  the  course. 

Finally,  consistent  with  their  positive  impressions  of  the  course,  almost  all  attendees  (98 
percent)  would  recommend  the  course  to  coworkers.  The  main  reason  that  they  would 
recommend  the  course  focused  on  the  team  building  content  of  the  course,  including  the 
communication  portion,  which  can  help  make  individuals  and  teams  be  more  effective  in  their 
jobs  and  improve  patient  care.  In  addition,  participants  highlighted  the  ability  of  DOM  to  change 
peoples’  mindsets.  They  also  pointed  out  that  participants  should  be  able  to  integrate  the  skills 
that  were  learned  into  their  jobs. 

Level  II,  III,  and  IV  data.  Data  concerning  the  development  and  evaluation  of  DOM  beyond 
trainee  reactions  are  limited.  As  of  January  2003,  over  160  surgical  staff  members  at  Methodist 
University  Hospital  in  Memphis,  TN,  had  completed  DOM  training.  An  evaluation  of  DOM  at 
Methodist  Hospital  found  improvements  in  participants’  attitudes  toward  the  importance  of 
teamwork  issues  in  the  OR,  favorable  reactions  concerning  the  usefulness  of  DOM  training,  and 
a  50  percent  reduction  in  the  number  of  surgical  count  errors.  However,  the  small  sample  size 
makes  it  difficult  to  assess  the  generalizability  of  the  results.  Moreover,  the  lack  of  control 
groups  makes  it  difficult  to  determine  whether  the  training  caused  these  improved  outcomes. 
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Table  7.  Posttraining  opinions  about  Dynamic  Outcomes  Management^. 


Total  (n=78) 

Item 

Mean 

Std. 

Dev. 

Agreement 

Neutral 

Disagreement 

1  am  confident  that  1  understood 
the  training  content 

4.6 

.65 

98% 

1% 

1% 

The  training  was  well  organized. 

4.7 

.72 

97% 

f 

3% 

1  am  confident  that  1  can  use  the 
knowledge  that  1  learned  on  the 
job. 

4.6 

.68 

96% 

3% 

1% 

1  am  confident  that  1  can  perform 
the  tasks  that  were  trained. 

4.4 

.70 

95% 

4% 

1% 

As  a  result  of  this  training,  1  feel 
more  confident  about  my  ability  to 
work  effectively  In  a  team. 

4.5 

.73 

92% 

7% 

1% 

The  training  content  was 
appropriate  for  my  department. 

4.4 

.83 

91% 

6% 

3% 

Training  prepared  me  to  work 
effectively  in  my  job. 

4.4 

.78 

88% 

11% 

1% 

Training  was  an  effective  use  of 
my  time. 

4.4 

.86 

88% 

9% 

3% 

Training  will  help  my  department 
Improve  patient  safety. 

4.4 

.83 

87% 

10% 

3% 

Strengths  and  Weaknesses 

Strengths.  Our  review  of  the  literature,  observations  of  the  classroom  phase,  and  post 
training  assessment  suggest  that  participants  had  positive  reactions  to  this  training.  Participants 
indicated  that  the  training  was  well  organized,  and  they  felt  that  they  could  use  many  of  the 
strategies  discussed  during  training  upon  returning  to  their  jobs.  Second,  DOM  instructors  were 
extremely  professional  and  conducted  high-quality  training.  Although  not  necessarily  the  most 
practical  approach  to  implementing  training  throughout  a  large-scale  organization  like  the  DoD, 
using  professional  instructors  from  the  course  vendor  resulted  in  significantly  better  and 
consistent  instruction.  Third,  DOM  staff  relayed  to  us  that  there  was  no  delay  (like 
MedTeams™)  between  the  classroom  phase  of  DOM  and  when  the  safety  tools  are  implemented. 
Therefore,  skill  decay  is  less  likely  with  this  program.  Finally,  the  DOM  developers  are 
beginning  to  make  efforts  to  collect  additional  data  on  DOM  effectiveness  beyond  trainee 
reactions.  Pre-  and  posttraining  attitude  data  are  currently  being  collected  and  discussions  with 
the  developer  indicated  that  future  studies  are  planned  to  examine  DOM  effectiveness.  The 
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results  from  the  Methodist  Hospital  Investigation,  which  showed  a  reduction  in  sponge  count 
errors,  are  encouraging. 

Weaknesses.  Nevertheless,  DOM,  like  the  other  programs,  does  have  its  limitations.  First, 
we  could  uncover  no  evidence  that  the  results  of  an  in-depth  pretraining  needs  analysis  drove  the 
development  of  DOM.  It  is  our  impression  that  the  course  developer  extracted  this  information 
from  CRM  training  and  subject  matter  experts  customized  the  materials  to  health  care.  CTI  staff 
then  visit  the  hospital  in  which  training  will  be  implemented  to  make  any  additional 
modifications  to  the  courseware  that  is  required.  Second,  like  MedTeams  and  MTM,  a  primary 
objective  of  DOM  training  is  to  develop  team  skills.  However,  most  of  the  classroom  instruction 
focused  on  mastering  declarative  and  some  procedural  knowledge.  There  wasn’t  much  time 
devoted  to  skills  practice.  Third,  DOM  did  not  employ  a  cultural  assessment/evaluation 
component  prior  to  implementing  the  training.  As  a  result,  it  is  entirely  possible  that  DOM  is 
effective  only  in  hospitals  that  have  already  made  a  commitment  to  patient  safety.  The  kickoff  by 
the  Associate  Dean  emphasized  the  medical  center’s  commitment  to  DOM  and  patient  safety. 
Also,  the  pretraining  data  suggested  that  the  organization  has  a  culture  that  supports  teamwork. 
Finally,  although  we  did  not  specifically  collect  this  data,  the  costs  of  implementing  DOM  are 
likely  to  be  higher  that  MedTeams™  and  MTM.  This  is  primarily  a  function  of  the  fact  that  CTI 
relies  on  its  own  cadre  of  instructors  to  conduct  training  and  a  full  array  of  consultative  services. 
While  this  produces  reliable,  high-quality  instructors,  we  question  the  viability  of  such  a  strategy 
when  training  must  be  delivered  in  a  timely  fashion  to  multiple  hospitals. 

Summary 

In  conclusion,  the  DOM  course  was  extremely  well  received.  There  was  great  support  for 
DOM  training  and  a  strong  commitment  to  patient  safety  by  that  organization.  The  content 
covered  important  aspects  of  teamwork  and  presented  similar  strategies  to  those  discussed  during 
MedTeams™  and  MTM  training.  In  our  opinion,  the  quality  of  instruction  was  perhaps  the  best 
of  the  three  programs  we  reviewed;  however,  there  are  many  practical  limitations  with  using 
vendor  instructors  when  introducing  medical  team  training  in  the  DoD.  Finally,  we  could  not 
uncover  much  information  about  how  the  DOM  course  was  developed,  which  caused  us  some 
concern.  There  was  no  evidence  to  suggest  that  DOM  was  in  fact  targeting  the  right  skills  for 
development  or  that  the  training  objectives  were  appropriate. 
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Chapter  5.  Discussion* 

Introduction 

In  this  section,  we  summarize  the  results  from  our  case  study  analysis  of  DoD-sponsored 
medical  team  training  programs.  As  noted  earlier,  our  primary  goal  throughout  this  project  was 
to  provide  an  independent  and  object  assessment  of  MedTeams™,  MTM,  and  DOM  (now  called 
LifeWings'^),  based  on  the  information  we  had  access  to.  Because  the  vast  majority  of  this 
information  was  qualitative  in  nature,  we  have  purposely  refrained  from  making  comparative 
judgments  among  the  three  training  programs.  From  time  to  time,  we  have  pointed  out  strengths 
and  weaknesses  based  on  the  information  we  collected.  This  was  done  to  provide  the  DoD  with 
as  much  information  as  possible.  However,  most  of  the  programs’  differences  were  minor  in 
nature  and  overall  we  felt  that  these  programs  were  fairly  equivalent  for  addressing  the 
awareness  phase  of  training.  Therefore,  the  conclusions  and  recommendations  that  follow  apply 
equally  to  MedTeams™,  MTM,  and  DOM. 

Summary 

Are  the  Medical  Treatment  Facilities  Ready  for  Training? 

The  success  of  any  training  intervention  depends  not  only  on  the  training  content  and 
instructional  techniques,  but  also  on  how  the  intervention  is  positioned,  supported,  and 
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reinforced  by  the  organization.  In  particular,  congruence  between  the  objectives  of  training  and 
the  organization’s  safety  culture  is  critical  to  ensuring  the  transfer  of  trained  behaviors.  A  safety 
culture  is  defined  as  a  shared  belief  about  the  importance  of  safety,  which  leads  to  specific  safety 
norms  that  dictate  behavior  within  the  group.  Previous  research  by  Helmreich  and  colleagues^^’"^^ 
has  demonstrated  the  importance  of  a  positive  safety  culture  in  both  aviation  and  health  care. 

Despite  the  fact  that  publicly-available  instmments  have  been  developed  for  assessing  the 
safety  culture  of  health  care  organizations,  '  we  were  unable  to  locate  any  evidence  that 
MedTeams™,  MTM,  or  DOM  conducted  a  pretraining  safety  culture  assessment.  As  a  result,  it 
is  entirely  possible  that  CRM-derived  team  training  programs  only  work  in  medical  facilities  that 
have  already  made  a  commitment  to  teamwork,  secured  upper-level  management  support,  and 
recognized  the  need  for  change.  Therefore,  we  recommend  that  future  patient  safety  initiatives, 
whether  it’s  a  training  intervention  or  otherwise,  always  conduct  a  thorough  pretraining  safety 
culture  analysis  to  identify  barriers  that  can  mitigate  the  transfer  of  trained  skills. 

Is  Aviation  Crew  Resource  Management  the  Right 
Starting  Point  for  Medical  Team  Training? 

Several  researchers  have  recently  suggested  that  health  care  providers  look  to  CRM  in 
aviation  as  a  model  for  reducing  medical  errors.  The  argument  for  adopting  CRM  training  is 


*  Note:  Appendixes  cited  in  this  report  are  provided  electronically  at 
httD://www.ahra.QOv/aual/teamtrain/index.html#app . 

^  The  AHRQ  Hospital  Survey  of  Patient  Safety  Culture  (http://www.ahrq.gov/qual/hospculture/)  was  released  in 
September  2004,  after  the  work  on  this  study  was  completed,  and  thus  is  not  included  in  the  references. 
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based  on  several  important  similarities  between  medicine  and  aviation.  For  example,  both 
domains  require  multidisciplinary  teams  of  highly-trained  professionals  to  perform  complex  and 
dynamic  tasks;  both  are  conducted  in  high  risk  environments  where  the  consequences  of  error 
can  be  deadly;  and  both  require  the  team  members  to  make  decisions  based  on  limited 
information  under  conditions  of  high  stress,  high  workload,  and  time  pressure.  Moreover,  recent 
research  by  Helmreich  and  colleagues'^^  has  shown  that  pilots  and  health  care  professionals 
exhibit  similar  attitudes,  such  as  the  mistaken  perception  of  invulnerability  to  the  effects  of  stress 
and  fatigue.  Given  these  similarities,  it  is  not  surprising  that  all  three  programs  are  based  on  the 
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aviation  CRM  model.  Specifically,  MedTeams  is  based  on  the  U.S.  Army’s  CRM  training 
course  for  helicopter  crews;  Medical  Team  Management  (MTM)  is  based  on  the  U.S.  Air 
Force’s  CRM  training  program  for  aviators;  and  Dynamic  Outcomes  Management 
(DOMyLifeWings'^'^  is  based  on  military  CRM  training  for  fighter  and  cargo  pilots. 

Despite  these  similarities,  there  are  a  number  of  important  differences  between  aviation  and 
medicine.  For  example,  the  size  and  composition  of  teams  varies  greatly  between  the  two 
domains.  Whereas  commercial  flight  crews  include  2-3  pilots  whose  performance  is  largely 
dictated  by  standard  operating  procedures,  medical  teams  include  up  to  15  physicians,  assistants, 
nurses,  technicians,  and  staff  from  other  disciplines  whose  performance  is  less  amenable  to 
standardization.'^^  In  addition,  because  of  their  repetitive  and  highly  scheduled  nature,  flight 
operations  are  well-suited  to  certain  CRM  tools  such  as  prebriefings/debriefmgs,  checklists,  and 
quick  reference  cards.'^^  By  way  of  comparison,  certain  medical  operations — such  as  emergency 
medicine — typically  cannot  be  planned  in  advance.  This  may  make  them  ill-suited  to 
interventions  which  require  such  strict,  scheduled,  and  orderly  procedures. 

In  addition  to  these  differences,  it  is  important  to  note  that  CRM  is  not  a  universal  remedy. 
CRM  by  itself  will  not  eliminate  all  the  systematic  contributors  to  medical  error.  Rather  CRM  is 
one  component  of  a  comprehensive  approach  to  improving  patient  safety.  For  example,  in 
AHRQ’s  Evidence-based  Practice  Report  (Number  43),  Making  Health  Care  Safer:  A  Critical 
Analysis  of  Patient  Safety  Practice,  50  safe  patient  practices  or  areas  for  system  improvements 
were  identified,  one  of  which  was  CRM.  Furthermore,  this  report  noted  that  the  evidence  base 
for  CRM’s  effectiveness  was  viewed  as  low  relative  to  others,  but  the  report  also  suggests  further 
research  on  the  introduction  of  CRM  in  health  care  is  likely  to  be  beneficial. 

The  differences  between  aviation  and  health,  as  well  as  the  limited  evidence  base,  do  not 
negate  the  potential  value  of  introducing  CRM  training  in  medicine.  Rather,  they  highlight  the 
importance  of  carefully  tailoring  CRM  principles  and  practices  from  aviation  to  medicine  and 
the  need  for  future  empirical  investigations  that  demonstrate  CRM’s  effectiveness.  This  tailoring 
should  be  driven  by  the  results  of  a  thorough  needs  analysis. 

Have  the  Needs  Analyses  Gone  Deep  Enough? 

All  training  programs  should  begin  with  a  comprehensive  needs  analysis.  At  a  minimum,  this 
should  include  an  organizational  analysis  (to  identify  the  organization’s  readiness  for  change),  a 
personnel  analysis  (to  identify  the  workforce’s  specific  knowledge,  skill,  or  attitude 
deficiencies),  and  a  task  analysis  (to  develop  and  sequence  the  learning  objectives).'^^’^^ 

Presently,  at  least  two  of  the  three  major  team  training  programs — MTM  and 
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MedTeams  — ^began  with  at  least  a  partial  needs  analysis.  For  example,  MTM  was  developed 
as  a  result  of  ineffective  teamwork  and  communication  as  a  recurrent  theme  in  major  medical 
events.  A  subsequent  validation  of  the  need  occurred  with  a  review  of  60  critical  incidents, 
which  revealed  communication  problems  were  the  single  largest  contributing  cause  of  medical 
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errors.  Similarly,  MedTeams  began  by  a  systematic  review  of  closed  case  records,  which  was 
scored  using  a  teamwork  failure  checklist  to  identify  trends.^  However,  subsequent  versions  of 
MedTeams™  have  involved  customizing  the  original  ED  curriculum  to  L&D  and  the  OR  by 
adapting  the  case  studies  and  examples  to  the  clinical  context  with  the  aid  of  subject  matter 
specialists.  MTM  has  not  engaged  in  such  customization  and  has  advocated  a  one-size-flts-all 
approach.  DOM  has  employed  a  similar  approach  to  MedTeams™. 

We  question  whether  or  not  the  MedTeams™  and  DOM  strategy  will  adequately  address  the 
unique  team  requirements  of  different  medical  services.  Our  expert  panel  suggested  that 
variation  is  likely  so  we  recommend  additional,  deeper-level  analyses  when  customizing  medical 
team  training  to  a  specific  specialty  area.  Alternatively,  we  see  some  merit  in  the  MTM 
approach,  if  a  generic  set  of  skills  exists  that  cuts  across  all  medical  specialties.  However,  these 
skills  and  the  behaviors  they  represent  would  have  to  be  established  and  agreed  upon. 

Are  the  Current  Instructional  Strategies  Appropriate? 

Despite  improvements  in  training  technology,  our  case  study  analysis  found  that  all  three 
programs  rely  almost  exclusively  on  classroom-based  instructional  techniques — such  as  didactic 
lectures,  case  studies,  videotaped  vignettes,  and  pretraining  readings — to  deliver  their  course- 
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specific  content.  For  example,  MedTeams  includes  an  8-hour  block  of  classroom  instruction 
that  contains  an  introduction  module,  five  learning  modules,  and  an  integration  unit. 
Supplementary  materials  include  a  30-minute  video  that  depicts  examples  of  good  and  bad 
performance.  After  completing  the  classroom  training,  each  team  member  participates  in  a  4- 
hour  practicum  that  involves  practicing  teamwork  behaviors  and  receiving  feedback  from  a 
trained  instructor.  Coaching,  mentoring,  and  review  sessions  are  also  provided  during  regular 
work  shifts.^  Similar  instructional  strategies  are  currently  used  by  MTM  and  DOM.  Our  review 
of  the  published  literature  suggests  that  these  strategies  are  best  suited  to  developing  factual  or 
declarative  knowledge  of  CRM  principles,^  ^  such  as  that  which  occurs  during  awareness  training. 

To  date,  none  of  the  programs  use  simulators,  such  as  the  type  used  in  Anesthesia  Crisis 
Resource  Management  ^  to  provide  trainees  with  the  opportunity  for  skills  practice  with 
instructor  feedback,  as  recommended  in  aviation  CRM.  Although  both  classroom-based  and 
simulation-based  strategies  are  justifiable  vehicles  for  delivering  the  training  content,  the  sense 
that  we  gained  from  our  review  was  that  classroom-based  techniques  are  used  primarily  because 
they  have  always  been  used  in  the  past,  not  because  they  are  particularly  well-suited  to  specific 
teamwork-related  competencies.  Therefore,  we  recommend  that  future  research  in  MTT  address 
which  instructional  strategies  are  most  appropriate  for  which  type  of  competencies,  and  then 
tailor  the  training  such  that  the  instructional  strategies  are  chosen  to  specifically  complement  the 
training  content. 

Are  the  Current  Methods  for  Delivering  Ciassroom  Training 
Sufficient? 

Each  program  has  two  primary  instmctional  components:  a  classroom-based  (i.e., 
implementation)  component  and  post-classroom  (i.e.,  sustainment)  component.  In  this  section, 
we  address  the  classroom-based  component.  Our  document  review  revealed  that  both 
MedTeams™  and  MTM  use  a  “train-the-trainer”  paradigm,  whereby  individuals  from  the 
medical  treatment  facilities  receive  instruction  in  teamwork  skills,  and  then  return  to  their 
respective  organizations  to  train  the  rest  of  their  colleagues.  However,  our  review  also  suggested 
that  MedTeams™  and  MTM  have  few  formal  mechanisms  for  ensuring  consistency  and 
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preventing  performance  degradation  among  instructors  once  they  return  to  their  respective 
facilities.  To  some  extent,  this  issue  is  not  relevant  for  DOM,  because  a  small  cadre  of  trainers 
from  Crew  Training  International  (CTI)  conducts  all  the  training. 

TTVI 

For  example,  upon  successfully  completing  the  train-the-trainer  course,  MedTeams 
instructors  are  then  “certified”  by  Dynamics  Research  Corporation.  Unfortunately,  there  is  little 
publicly  available  information  about  this  certification  process.  For  example,  it  is  unclear  whether 
the  certification  is  based  on  the  instructors’  knowledge  of  teamwork  principles,  their  actual  level 
of  teamwork  skills  (e.g.,  communication,  decision-making,  etc.),  their  ability  to  teach  teamwork- 
related  issues  to  others,  or  some  combination  of  these  factors.  MTM  has  even  less  stringent 
procedures  for  ensuring  consistency  and  preventing  performance  degradation.  It  is  requested  (but 
not  required)  that  potential  MTM  instructors  have  at  least  5  years  of  clinical  experience  in  their 
specialty  areas,  at  least  1  year  of  time  remaining  in  the  Air  Force,  that  they  be  competent 
speakers,  and  that  they  have  previous  experience  delivering  training. 

The  presence  of  formal  procedures  for  ensuring  consistency  during  the  implementation  phase 
is  essential  to  the  success  of  any  training  program.  However,  our  observations  and  interviews 
suggest  that  the  implementation  of  MTT  programs  may  not  always  correspond  to  their 
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developers’  specifications.  For  example,  our  observation  of  the  MedTeams  course  showed  that 
the  classroom  portion  of  training  varied  from  instructor  to  instructor,  with  some  courses  lasting 
only  3  hours.  We  discovered  similar  results  while  interviewing  MTM  instructors. 

To  some  extent,  this  is  to  be  expected  with  the  train-the-trainer  paradigm.  Specifically,  the 
train-the-trainer  paradigm  is  designed  to  be  a  cost-effective  means  for  distributing  training  to  a 
large  number  of  people  within  a  short  period  of  time.  However,  the  trade-off  is  that  the  training 
developers  have  less  control  over  how  the  training  is  actually  implemented  at  the  host  sites. 

DOM  has  taken  an  alternate  approach.  By  using  a  small  cadre  of  trainers  from  Crew  Training 
International,  they  can  reach  a  much  smaller  audience;  however,  this  provides  them  with  greater 
control  over  the  quality  of  instruction.  Because  our  observations  and  interviews  were  based  on  a 
small,  nonrepresentative  sample,  it  is  impossible  to  estimate  the  actual  base  rate  of  instructor 
deviations  from  the  prescribed  training  syllabi.  Nevertheless,  we  caution  MTT  developers  to 
carefully  and  periodically  monitor  the  implementation  of  their  respective  MTT  programs  to 
ensure  that  the  training  is  delivered  as  designed. 

Are  the  Current  Methods  of  Sustainment  Sufficient? 

In  this  section,  we  address  the  post-classroom  component  of  training.  Our  document  review 
revealed  that  all  three  programs  have  some  form  of  post-classroom  follow-up.  During  this  time, 
trainees  are  provided  with  the  opportunity  to  practice  their  trained  skills,  and  to  receive  some 
degree  of  feedback  from  their  site  coordinator. 

For  example,  the  post-classroom  component  of  MedTeams  lasts  for  approximately  6 
months.  During  this  time,  trainees  are  encouraged  to  practice  their  newly-trained  skills. 

According  to  our  document  review,  MedTeams™  employs  a  number  of  tools  for  sustaining 
effective  team  performance,  such  as  trainees’  monitoring  one  another’s  performance.  In  addition, 
routine  team  meetings  are  conducted  to  ensure  team  members’  continued  focus;  status  boards  are 
used  to  maintain  team  members’  situational  awareness  about  particular  patients,  nurses  routinely 
participate  in  morbidity  and  mortality  (M&M)  meetings,  and  performance  evaluations  directly 
consider  teamwork  issues.^^  MTM  includes  similar  techniques  for  sustaining  and  reinforcing 
teamwork-related  issues  in  the  post-classroom  environment,  such  as  periodic  (scripted)  safety 
drills,  periodic  team  leader  meetings,  formal  recognition  of  improved/effective  teamwork  during 
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the  trainees’  annual  performance  reviews,  and  a  report  on  implementation  progress  to  the  Air 
Force  Patient  Safety  office.^^  DOM  also  includes  such  practice  with  the  customized  development 
of  checklists  and  other  tools  that  support  safety  practices. 

Previous  research  suggests  that  the  presence  of  formal  procedures  for  sustaining  trained 
behaviors  in  the  posttraining  environment  is  essential  for  ensuring  the  success  of  training.^  ^ 
However,  our  review  suggests  that  the  respective  course  developers  spent  considerably  more 
time  focused  on  the  short,  classroom-based  component  of  training  (which  typically  last  between 
8-12  hours)  than  on  the  extensive,  post-classroom  component  (which  can  last  up  to  6  months). 
This  is  ironic,  given  that  the  classroom  phase  was  focused  largely  on  the  awareness  of  teamwork- 
related  issues.  For  each  program,  the  classroom  component  covered  teamwork-related 
knowledge  and  attitudes;  there  was  substantially  less  opportunity  for  actual  skills  practice  with 
instructor  feedback.  Therefore,  we  urge  MTT  developers  to  pay  greater  attention  to  the  post¬ 
classroom  component  of  training,  because  this  is  where  the  real  changes  in  learning  (i.e., 
development  of  teamwork-related  skills),  performance  (i.e.,  greater  use  of  checklists,  pre- 
briefings,  cross-checking,  etc.)  and  results  (i.e.,  reductions  in  medical  errors)  are  likely  to  occur. 

Recommendations 

In  this  section,  we  provide  theory-  and  practice-based  recommendations  for  improving 
medical  team  training.  The  overarching  theme  in  this  section  is  the  need  for  a  more  thorough 
understanding  of  the  medical  team  performance  and  medical  team  training  domains  than 
currently  exists. 

standardize  the  Knowledge,  Skills,  and  Attitudes 

A  recent  review  by  Cannon-Bowers  and  colleagues^"  has  noted  that  the  team  skills  literature 
is  confusing,  contradictory,  and  plagued  with  inconsistent  labels  and  definitions.  In  some  cases, 
different  labels  are  often  used  to  describe  the  same  teamwork  competencies;  in  other  cases,  the 
same  labels  are  used  to  describe  different  competencies.  Thus,  our  first  recommendation  is  to 
develop  a  standard  nomenclature  that  names  and  defines  the  core  teamwork-related  knowledge, 
skills,  and  attitude  competencies  for  successful  teamwork  in  health  care. 

We  envision  this  first  taxonomy  as  all-inclusive,  in  the  sense  that  it  would  incorporate  the 
core  competencies  that  potentially  influence  all  types  of  medical  teams,  regardless  of  specialty  or 
context.  In  other  words,  the  proposed  taxonomy  would  be  medical-team  generic.  This  medical- 
team  generic  specification  connotes  two  theory-driven  steps:  determining  an  appropriate  level  of 
explanation  for  identifying  core  teamwork  competencies  and  determining  which  of  these 
competencies  are  relevant  to  medical  teams. 

The  first  step,  determining  an  appropriate  level  of  explanation,  is  necessary  to  ensure  that  the 
constructs  included  in  the  taxonomy  are  conceptualized  broadly  enough  to  span  the  entire  health 
care  field,  yet  specific  enough  to  facilitate  valid  measurement.  Further,  although  this  list  of  core 
competencies  should  reflect  all  relevant  aspects  of  team  performance,  it  must  be  concise  enough 
to  generate  teamwork  and  team  training  research  and  to  facilitate  team  training  needs  analyses  in 
organizations. 

The  second  step,  determining  relevant  core  competencies,  encompasses  two  activities.  One  is 
to  establish  which  of  the  many  competencies  manifested  in  previous  research  are  relevant  to 
virtually  all  medical  teams;  a  second,  and  perhaps  more  demanding  task  is  to  identify  core 
medical-team  competencies  that  have  not  emerged  from  team  research  in  other  domains.  For 
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guidance  in  this  area,  researchers  might  rely,  to  some  extent,  upon  medical  experts  like  those 
engaged  in  the  panel  that  AIR  held  in  January  2003  (refer  to  Appendix  D  for  a  list  of  panel 
participants). 

However,  we  believe  that,  along  with  developing  a  theory  of  medical  team  performance, 
standard  job-analytic  techniques,  such  as  task  questionnaires,  structured  interviews,  and  non- 
obtrusive  observations  will  yield  the  most  valid  information.  Moreover,  we  emphasize  the 
importance  of  large-scale,  stratified  data  collections  because  the  goal  is  to  identify  generic 
competency  requirements  with  which  the  medical  community  at  large  concurs. 

Identify  Practice-Specific  Training  Requirements 

We  believe  that  the  medical-team  generic  taxonomy  described  above  would  be  useful  to  both 
theoretical  and  applied  research;  nevertheless,  we  believe  that  no  single  set  of  team  knowledge 
and  skills  can  be  applied  equally  across  all  medical  practices  and  contexts.  For  purposes  of  this 
discussion,  we  define  a  “practice”  as  a  medical  specialty  or  subspecialty,  such  as  emergency 
medicine,  general  or  family  medicine,  intensive  care,  surgical  medicine,  obstetrics,  and  so  forth. 
Medical  practices  differ  dramatically  across  a  variety  of  criteria,  including  team  size,  lifespan, 
acuteness  (i.e.,  consequence  of  error),  and  member  composition,  to  name  but  a  few.  As  an 
example,  emergency  medicine  providers  function  in  hospital  emergency  departments,  in 
emergency-response  mobile  units,  and  on  battlefields.  Similarly,  urban  and  rural  general 
practitioners  operate  in  independent  or  multipractitioner  offices,  as  well  as  in  community  walk-in 
clinics.  Neither  the  competencies  that  impel  successful  teamwork  nor  an  optimal  team  training 
strategy  can  be  expected  to  generalize  across  these  diverse  contexts.  And,  of  course,  not  all 
members  within  the  same  team  will  necessarily  need  the  same  knowledge,  skills,  and  attitudes. 

Thus,  our  second  recommendation  is  to  identify  practice-specific  training  requirements  for 
diverse  health  care  contexts.  These  taxonomies  would  not  be  redundant  with  the  generic 
taxonomy.  Rather,  a  practice-specific  taxonomy  would  reflect  “core”  competencies  in  the  sense 
of  denoting  the  knowledge,  skill,  and  attitude  requirements  that  are  central  to  teamwork  in  a 
given  practice.  The  medical  content  and  procedures  that  define  that  practice  would  drive  the 
identification  of  relevant  team-competency  requirements. 

Virtually  no  previous  research  has  addressed  the  manner  in  which  differences  within  and 
between  medical  practices  should  be  reflected  in  practice-specific  taxonomies.  Yet  we  find  this 
issue  sufficiently  compelling  to  warrant  further  investigation.  Because  these  taxonomies  are 
derived  from  the  medical  characteristics  of  specific  practices  (and  the  contexts  within  them), 
subject-matter  experts  who  represent  each  practice  might  be  invaluable  in  identifying  practice- 
specific  team  competencies  that  are  not  redundant  with  the  generic  taxonomy.  Nevertheless,  we 
would  also  suggest  that  researchers  avail  themselves  of  job  analysis  questionnaires,  structured 
interviews,  and  nonobtrusive  observations. 

Leverage  Existing  Knowledge  About  Teamwork  Training 

Although  the  field  of  medical  team  training  is  still  in  its  infancy,  the  science  and  practice  of 
team  training  is  far  from  new.  Research  on  the  design,  implementation,  and  evaluation  of  team 
training  programs  began  nearly  50  years  ago.  Since  that  time,  a  substantial  body  of  research  has 
been  conducted  regarding  the  most  effective  strategies  and  techniques  for  training  teamwork- 
related  knowledge,  skills,  and  attitudes.  For  example,  Salas  and  his  colleagues  have  compiled  an 
extensive  collection  of  principles  and  guidelines  for  assertiveness  training,^^  cross-training,^^ 
stress  management  training,^^  and  team  self-correction.^^ 
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Unfortunately,  the  existing  medical  team  training  programs  do  not  appear  to  have  leveraged 
this  body  of  research.  For  example,  as  noted  earlier,  the  MTT  programs  that  we  reviewed  rely 
almost  exclusively  on  classroom-based  training  methods,  rather  than  choosing  from  a  variety  of 
instructional  strategies  to  complement  the  specific  training  content.  With  few  exceptions,  new 
advances  in  training  technology — such  as  computer-based  training,  low-fidelity  simulations, 
embedded  training,  and  scenario-based  training — have  rarely  been  used,  despite  growing 
evidence  regarding  their  effectiveness.^^  Recent  advances  in  training  theory — such  as  the  effect 
of  pre-  and  posttraining  factors  on  training  outcomes,  the  effect  of  practice  schedules  on  skill 
acquisition  and  retention,  and  the  critical  role  of  individual  differences  in  shaping  trainees’ 
motivation — have  similarly  been  ignored.  Each  of  these  factors  has  been  shown  to 

improve  the  effectiveness  of  team  training  programs.  Thus,  our  third  recommendation  is  that 
instructional  designers  leverage  such  information  in  their  own  medical  team  training  programs. 

Develop  a  Standardized  Training  Specification 

_ _ 

Two  different  MTT  programs  are  currently  sponsored  within  the  DoD:  MedTeams  and 
MTM  (DOM  is  not  funded  by  the  DoD  but  has  been  implemented  at  one  naval  medical  hospital). 
As  we  have  shown  in  previous  sections,  these  programs  use  similar  instructional  strategies,  have 
similar  training  content,  and  have  demonstrated  similar  results  concerning  their  effectiveness. 
These  similarities  beg  the  question:  Why  have  three  separate  MTT  programs?  Despite  outward 
appearances,  this  is  not  a  trivial  question.  Practically  speaking,  multiple  training  programs  could 
be  problematic,  especially  during  deployment  when  intact  teams  are  split  up.  For  example,  a 
deployed  team  may  consist  of  some  individuals  who  have  received  MTT  and  some  who  have 
not.  Alternatively,  a  deployed  team  may  include  staff  from  multiple  branches  of  the  armed 
forces,  each  of  which  may  use  a  different  MTT  program.  This  could  create  a  great  deal  of 
confusion,  for  example  if  the  team  members  were  practicing  similar  teamwork  behaviors,  but 
calling  them  by  different  names  or  using  different  approaches  to  accomplish  the  same  task. 

Standardizing  MTT  would  produce  greater  consistency  in  the  design,  delivery,  and 
evaluation.  Thus,  our  fourth  recommendation  is  that  the  DoD  develops  a  training  specification 
for  MTT  programs  in  specific  practice  areas.  In  addition  to  describing  the  core  teamwork 
competencies  for  each  practice  area,  the  specification  would  outline  the  appropriate  instructional 
strategies  for  each  core  competency,  the  appropriate  sequencing  of  training  activities,  and 
outcome  measures  for  assessing  the  degree  of  skill  acquisition,  to  name  but  a  few.  Moreover, 
such  a  training  specification  would  allow  the  DoD  substantial  flexibility  in  procuring  and 
administering  the  training.  For  example,  training  could  be  administered  internally  or  with  the  aid 
of  contractors,  depending  on  time  and  resource  constraints.  Regardless  of  who  offers  the  training, 
the  content  and  delivery  would  be  essentially  standardized  by  the  program  specification. 

Develop  Technical  Assistance  on  Crew  Resource 
Management  Issues  in  Health  Care 

The  Federal  Aviation  Administration  (FAA)  promotes  aviation  safety  by  two  primary 
mechanisms:  education  and  enforcement.  Of  the  two,  education  is  the  most  proactive  way  to 
prevent  problems  from  occurring.  One  of  the  FAA’s  primary  means  of  educating  the  aviation 
community  on  important  initiatives  is  through  the  use  of  advisory  circulars  (ACs).  In  general,  ACs 
present  guidelines  for  developing,  implementing,  and  evaluating  various  FAA  initiatives  such  as 
pilot  Crew  Resource  Management  (CRM)  and  simulation-based  training  programs. 
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Advisory  circulars  are  designed  to  be  informative.  They  typically  present  one  or  more 
ways — ^but  not  the  only  way — of  addressing  a  particular  issue.  For  example,  the  FAA’s  advisory 
circular  on  CRM  training  includes  a  definition  of  CRM  concepts,  fundamentals  of  CRJM  training 
implementation,  suggested  curriculum  topics,  guidelines  for  assessing  the  effectiveness  of  CRM 
training,  example  behavioral  markers  for  specific  CRM  skills,  and  suggestions  for  further 
reading. 

Under  the  Patient  Safety  and  Quality  Improvement  Act  of  2005  (P.L.  109-41),  the  Agency 
for  Healthcare  Research  and  Quality  (AHRQ)  has  been  charged  by  the  Secretary  of  HHS  with — 
among  other  things — ^providing  technical  assistance  to  the  newly  created  Patient  Safety 
Organizations  (PSOs)  that  will  support  providers  in  the  “improvement  of  patient  safety  and 
reduce  the  incidence  of  events  that  adversely  effect  patient  safety.”  We  recommend  that  the 
Agency  prepare  technical  assistance  documents  comparable  to  the  FAA  Advisory  Circulars  to 
help  the  PSOs  guide  providers  on  issues  related  to  team  training  and  error  prevention.  We 
believe  that  human  factors-related  technical  assistance  to  the  PSOs  would  go  a  long  way  to 
educate  the  medical  community  about  the  importance  of  MTT  for  ensuring  patient  safety,  and  for 
ensuring  consistency  across  MTT  programs. 

Conclusions 

In  conclusion  this  report  presents  an  in-depth  case  study  analysis  of  three  medical  team 
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training  programs,  MedTeams  ,  Medical  Team  Management,  and  Dynamic  Outcomes 
Management®.  This  was  the  first  independent  assessment  of  these  programs.  The  case  study 
approach  allowed  us  to  collect  detailed,  comprehensive  information  on  each  program,  which  we 
reported  along  a  common  set  of  variables.  Although  this  study  was  qualitative  in  nature,  it  is  the 
first  effort  to  capture  that  state-of-the  art  in  medical  team  training. 

AHRQ’s  Evidence  Report  43  (Making  Health  Care  Safer:  A  Critical  Analysis  of  Patient 
Safety  Practices)^^  suggested  that  future  research  on  medical  team  training  is  likely  to  be 
beneficial  and  have  a  significant  impact  on  patient  safety.  We  view  the  results  presented  in  this 
investigation  as  a  starting  point  for  future  studies  on  medical  team  training.  Here,  we  have 
provided  information  on  current  objectives,  strategies  and  successes  of  existing  programs  as  well 
as  where  opportunities  for  improvements  exist.  We  have  also  delineated  several  areas  where 
future  research  is  most  warranted.  However,  empirically-based  research  will  require  a  mandate 
from  program  sponsors,  Federal  agencies,  or  the  health  services  research  community;  greater 
access  to  health  care  workers  and  patients  to  collect  both  process  and  outcome  data;  and 
significant  resources  in  terms  of  time,  money,  and  personnel.  Nonetheless,  we  believe  that  such 
investments  are  worthwhile,  because  few  would  dispute  the  relation  between  team  performance 
and  safety.  The  challenge  is  to  show  irrefutable  evidence  that  substantiates  the  relation  between 
teamwork  in  health  care  and  the  desired  outcome,  a  reduction  in  errors,  because  the  medical  error 
rate,  although  unacceptably  high,  has  a  relatively  low  base  rate.  We  believe  that  this  can  be 
accomplished  under  the  right  conditions  and  point  to  the  on-going  L&D  study  as  an  example  of 
the  kind  of  investigations  that  are  required. 

In  summary,  we  believe  that  the  future  is  bright  for  medical  team  training  because  there  is  an 
existing  knowledge  base  from  aviation  and  other  high-risk  industries  on  which  health  care  can 
gain  traction.  Great  strides  have  been  made  with  the  introduction  of  the  development  and 
introduction  of  the  three  training  programs  described  in  this  report.  We  view  these  programs  as 
the  first  generation  of  what  we  believe  will  be  continuous,  sustained  advancements  in  medical 
team  training  over  the  next  decade  and  beyond. 
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